Also 7 days ago[1][2][3] but no upvote love so far. Which is curious given the (possibly short term, until these images join training corpus?) privacy benefits
It might be possible that these people have tried on more platforms (Face recognition APIs) but only reported those where they got good accuracy in terms of defeating system.
I personally would like to see tests done on facebook by uploading these images and checking if it can recognize it.
I was immediately was curious how it would protect against image compression and/or otherwise de-noising these protection pixel changes. Their paper does address this question, and for those that are interested:
Even image compression cannot defeat our cloak.
We use progressive JPEG [57], reportedly used
by Facebook and Twitter, to compress the images
in our dataset. The image quality, as standard
by Independent JPEG Group [1], ranges from 5 to
95 (lower value = higher compression). As shown
in Figure 15, image compression decreases the
protection success rate, but more significantly
degrades normal classification accuracy.
I wonder how it works, but it works.
p.s. Mobile friendly copy-paste: "Even image compression cannot defeat our cloak. We use progressive JPEG [57], reportedly used by Facebook and Twitter, to compress the images in our dataset. The image quality, as standard by Independent JPEG Group [1], ranges from 5 to 95 (lower value = higher compression). As shown in Figure 15, image compression decreases the protection success rate, but more significantly degrades normal classification accuracy."
They are lucky in this case because compression does defeat the cloak to some degree— However, compression at those levels also defeats the ability of Facebook to accurate identify you.
> However,we find that none of these transformations defeat our cloaks. The protection success rate remains 100% even when data augmentation is applied to cloaked images5. Ap-plying Gaussian blurring degrades normal accuracy by up to18% (as kernel size increases) while cloak protection success rate remains>98% (see Figure13). Adding Gaussian noise to images merely disrupts normal classification accuracy –the cloak protection success rate remains above 100% as the standard deviation of the noise distribution increases(seeFigure14). Even image compression cannot defeat our cloak.We use progressive JPEG [57], reportedly used by Facebookand Twitter, to compress the images in our dataset. The im-age quality, as standard by Independent JPEG Group [1],ranges from 5 to 95 (lower value = higher compression). As shown in Figure15, image compression decreases the pro-tection success rate, but more significantly degrades normal classification accuracy.
You don't even have to read the paper, it's in the FAQs:
"Can't you just apply some filter, or compression, or blurring algorithm, or add some noise to the image to destroy image cloaks?"
Short answer: No not really. Long answer: Look at the FAQs :)
They create a picture that is designed to fool facial recognition systems and then they will test it against JPEG and other image compression techniques and then run the facial recognition on the compressed version and if both of them pass then they don't change the image.
If the procedure fails then the GAN will know this and it would change the output so that it would pass both outcomes.
It doesn't use a GAN. I haven't read the paper closely, but is uses a feature extracting network and then tries to modify the features to match a different person. It doesn't try to take advantage of degenerative states in existing networks.
I think it does things like subtly change the shape of your eyes and whatnot. That makes people consistently difficult to pick out of a large set of unmodified faces, but a hu man still finds the images recognizable because they're comparing it to a smaller set.
They didn't create the model by training it on the facial recognizers they tested against, and they tested with several different feature extractors.
Stupid thought - technically couldn't any implementation include some sort of deliberate covert stenographic key? Granted that would likely be more "narrow down the implementation" without it being detectable via hashes per instance output.
While this seems to work against several current techniques, there's no guarantee it will work against all of them. It also offers no guarantees against future developments, and anything you put on the public internet is likely to be retained forever. Because of this I'd consider it an interesting proof of concept, but not something anyone should use as a privacy tool. You could consider it in cases where you're forced to provide a picture, for instance my public transport card requires one.
Color me extremely skeptical. A low-pass filter will make short work of any "tiny, pixel-level" changes designed to thwart ML. After all, one of the most tell-tale identifiers (space between eyes/nose/mouth) is still plainly observable and unaltered in the "cloaked" image.
If a human's neural network can correctly correlate the before/after examples, so can a computer's. They might have found an issue with some modern implementations of facial recognition, sure. But it's a false sense of security to claim "when someone tries to identify you using an unaltered image of you [...] they will fail."
Reproducing the image in this way is essentially a manual low-pass filter (although with little control over the parameters), so it's certainly one valid data point with which to test the hypothesis.
This was my first impression as well, except with less knowledge on the subject.
Someone, somewhere said "huh...", and placed another filter into the pipeline to handle these types of images.
While looking for something to sound smart wrt the tank-training myth, I found this interesting page: https://www.gwern.net/Tanks "The Neural Net Tank Urban Legend"
And interestingly, looking at the link for "superresolution needing learned downscalers" found this: https://arxiv.org/abs/1907.12904 "Learned Image Downscaling for Upscaling using Content Adaptive Resampler", code available at https://github.com/sunwj/CAR
So, IDK, seems like this Fawkes approach will be an interesting paper.
> If a human's neural network can correctly correlate the before/after examples, so can a computer's.
color _me_ skeptical, but
this is like saying we have functioning AGI; that artificial NNs are the same as the ones we have in our skulls. This to me, is an effect of the over-anthropomorphization of machine learning. It's a bad intuition to have.
However, I do agree. This is just one step in an arms race, and one iteration from being worthless.
That bit was more of a forward-looking statement about the future capabilities of image recognition, but yes it is somewhat hyperbolic in the general case. I don't believe we'll ever achieve AGI, but I do believe we'll have super reliable application-specific classifiers that vastly outperform humans and won't be fooled by tricks like this.
Q: Can't you just apply some filter, or compression, or blurring algorithm, or add some noise to the image to destroy image cloaks?
A: As counterintuitive as this may be, the high level answer is no simple tools work to destroy the perturbation that form image cloaks. To make sense of this, it helps to first understand that Fawkes does not use high-intensity pixels, or rely on bright patterns to distort the classification value of the image in the feature space. It is a precisely computed combination of a number of pixels that do not easily stand out, that produce the distortion in the feature space. If you're interested in seeing some details, we encourage you to take a look at the technical paper (also linked above). In it we present detailed experimental results showing how robust Fawkes is to things like image compression and distortion/noise injection. The quick takeaway is that as you increase the magnitude of these noisy disruptions to the image, protection of image cloaking does fall, but slower than normal image classification accuracy. Translated: Yes, it is possible to add noise and distortions at a high enough level to distort image cloaks. But such distortions will hurt normal classification far more and faster. By the time a distortion is large enough to break cloaking, it has already broken normal image classification and made the image useless for facial recognition.
It was pretty frustrating that they did not readily offer any example images for inspection, so against my better judgement I downloaded their binaries to run some experiements.
First, a source image at an approximate resolution that you might find on a social networking site: https://imgur.com/a/9szcC1m
I applied a difference filter between the two images in Photoshop, to show an example of the actual pertubations performed: https://imgur.com/a/q4zC7Ms
Since it's hard to see, I compressed the output to highlight what the program actually changed. It does seem like there is a good amount of disturbance to the image: https://imgur.com/a/1Sx68o3
Now, the real test. First, a Google reverse image search for the original file - identification is pretty bang-on: https://imgur.com/a/5HJwjPx
The only difference I'm seeing is a few images that are one or two images swapped in the "visually similar images" category.
So, I figured that that's the "best case" for the cloaked image - giving the search algorithm the full, unfiltered data, and the program still failed to disguise it. For fun, I thought I would use a "low-pass filter" (Google Lens pointed at my computer screen) as well, just for thoroughness. And the result surprised me!
So, it would seem that the algorithm's distortions more effectively come through in worse quality images! But, based on my full-resolution result, I wouldn't trust it to disguise something that is being directly uploaded to a social network.
Now, one important note is that reverse image search is probably not using a facial recognition model, but more like image chunk hashing - although I would also consider that something a privacy tool should defend against, which is why I included it.
All in all, very interesting and thanks for convincing me that I should actually test it out.
There's also a chance Google image search is looking at the filename of the image to get a bit more context. Does the reverse image search of the cloaked image still work if you rename it something unrelated?
This time, Google reverse image search did better at identifying the name of the singer in the cloaked image, instead of just giving the band name for the uncloaked.
Not super scientific since we don't really know what's going on behind-the-scenes with Google reverse image search, but it's certainly one adversary that doesn't seem to be easily fooled if there are other images of "you" out there for it to find. I also tried these small crops in Google Lens with less success (I got unrelated portraits for both images, cloaked or not).
The goal, afaik, is to stop facial detection software from learning to recognize you and put a face to a name, not to frustrate visual similarity searches. The images are supposed to be visually similar -- so similar that they're indistinguishable to a human viewer.
Surveillance software that purports to accurately identify a person across multiple images is not just looking for the same content with some visually insignificant modifications. It's reading your facial structure, attaching your name to it, and searching for it in every image received. Fawkes is working to defeat that specific use case, not all fuzzy matchers in general.
P.S. If you have a human assailant running a reverse image search for photos of you, I think you're well past the point that something like this could be expected to help.
But this suggests a way to defeat the cloak -- run your input through image similarity search, then run your facial recognition software on the hits. This won't work in full generality, not every picture is on the internet like that, but it can certainly help, I imagine.
I think that technique probably works a lot better with high-resolution professional headshots than it does with candid photos at the family reunion, for example.
However, if someone is willing to go to that level of effort, the target probably needs to aim for something a little more forceful than tricking Facebook's autotagger.
GIS is specifically designed to be good at finding similar images so it's going to work great for your test case. Facial recognition algorithms are solving a different problem.
If they were promising that cloaking would work well on GIS, that'd be a different matter. I can imagine wanting your images to not show up on GIS (because people would use them to try and find the source image on your profile, or something) but it's a different set of constraints at that point.
For cloaking a big use case would be "I took a selfie with a friend and want to share it on my instagram" and your goal is for that instagram selfie to not automatically connect with, for example, a surveillance photo of you at a protest. GIS is obviously not relevant to that scenario.
When I look at the cloaked image it seems that the tool is doing something legit. It's actually changing the shape of the face ever so slightly, rather than playing tricks with noise.
I echo the sentiment of other posters regarding reverse-image search. The original image should not be available to match against. That would be operator failure.
Haven't read the paper yet but sure will do. I wonder how it works, that so many face rec implementations are fooled but I don't know how similar they work. I would have guessed they know which features are extracted and modify the relevant regions.
I guess that face rec software will quickly adapt though. That said, we have invisible watermarks that are very resistant to compression or other filters.
Great concept, as long as the subject can avoid canonical image-to-name mappings such as airports (now scanning everyone), US’s REAL ID database, and the like.
That said, given that for most people the threat model is social or work rather than legal, something like this would be terrific to build into consumer insta-photo devices.
Certainly another tool in the privacy toolkit if you absolutely must surrender your likeness to someone else’s computer, but worth bearing in mind that this does not provide (and doesn’t purport to provide) the kind of privacy that strong encryption (or better yet, absent data) can provide.
Or better yet, burning Google and Facebook to the ground.
Technical solutions have never solved this sort of societal problem. Expecting a few individuals to fight against massive institutions with a little clever math is not going to work.
I can't help but roll my eyes at the the introduction's "unregulated facial recognition software" part of the introduction. That is a meaningless term given the lack of regulation in the first place examples and says in itself nothing about the effectiveness. The "Clipper Chips" infamous Skipjack was regulated. It annoys me mostly because meaningless rhetoric looks like they have no defensible stance.
That rant aside I am curious if this technique will lead to more resilent facial recognition and image parsing techniques to find the shape. Obviously the fact humans can still recognize it is a hint there is some other algorithim possible.
This is tested on existing models/Face Recognition API which means locked pre-trained models. So, They might have learned way to add pixels such that model outputs very different embedding. This is know issue in deep learning [0][1][2].
I believe, Model trained on cloaked images would defeat its purpose and make this technique useless.
[0] Su, Jiawei, Danilo Vasconcellos Vargas, and Kouichi Sakurai. "One pixel attack for fooling deep neural networks." IEEE Transactions on Evolutionary Computation 23.5 (2019): 828-841.
[1] Guo, Chuan, et al. "Countering adversarial images using input transformations." arXiv preprint arXiv:1711.00117 (2017).
[2] Liu, Yanpei, et al. "Delving into transferable adversarial examples and black-box attacks." arXiv preprint arXiv:1611.02770 (2016).
But the model will eventually be updated to detect and process the new cloaking images. So, to stay ahead, you decide to create a model that automatically generates different cloaking images, and... The whole system is now just a GAN : https://en.wikipedia.org/wiki/Generative_adversarial_network
I think there's a (hopefully strongly privacy preserving) combinatorial explosion here though. If current models can be trained to accurately-enough recognise me with, say, 100 training images - this tool might produce unique enough perturbations to require 100 images for each of the possible perturbations, potentially requiring you to train your new model using tens of thousands or millions of cloaked versions of the 100 images for each of the targets in your training set.
(If I were these researchers I'd totally be reaching out to AWS/Azure/GCE for additional research funding... <smirk>)
Not necessarily, because the changes are destructive. They can't restore what was there before, and they can't necessarily infer which image was cloaked and which was not.
If you use a new cloaking image for each picture you upload to social then they will all be embedded in a different location for a given feature extractor and an adversary wouldn’t be able to reverse search for linked pictures—that’s at least my understanding of how the method would need to be used. But if you keep using the same cloaking image, your adversary could definitely learn that process and effectively undo it.
The FAQ there addresses that, suggesting you can "dilute down" the ratio of normal-to-cloaked images in the public data sets the model creators train on, and hence reduce their future accuracy.
(So now you just need to somehow get as many cloaked photos of yourself uploaded and tagged to FB as they've collected in the last decade or so...)
"when someone tries to identify you using an unaltered image of you [...] they will fail."
I wonder how this holds up when someone takes a photo of that 'protected image'. I can imagine that if these miniscule pixel-scaled changes aren't visible to the naked eye, my crappy 6 megapixel camera will overlook it as well. If I then proceed to feed that image into my image recognition algorithm, is it still protected?
They go over the effects of compression - which they say only degrades the protection - but at the same time also degrades the identification accuracy of the AI model.
So if your crappy 6 megapixel camera cannot take a clear shot of the cloaked pixels - or effectively applying a blur filter - would also affect the AI detection.
More importantly, assuming they have a database of such cloaked images, what if someone just applies the same cloaking technique to the image of you? Can they still identify you?
That's making a pretty lazy assumption that even a quick read of the original article leads me to be sure it's incorrect.
There's quite a lot of comments here that stink of Dunning Kruger candidates, who read the headline and first paragraph, then just started typing their random "wisdom" assuming they're smarter and better informed that the team of PHD researchers who wrote the paper being discussed. (Am I just overly grumpy and judgemental today? Was HN always this bad?)
I can't speak for how effective Fawkes is but I can speak for the process. I just tried this out with 4 images.
One thing that I took notice of was how long the program ran on my computer. It took about 5 and a half minutes to obfuscate 4 images on an i9-9900K with the cpu was pegged at 100% the entire time. I can't imagine how long this would take on a low end laptop: especially if I needed to cloak a lot of images in bulk.
Another thing I noticed is that the discoloration that is applied to the final images can be easily mistaken for bruising. If I were to see someones post on social media and they looked like my results I'd be inclined to think that the poster is recovering from a bad fight or is a victim of abuse.
Other than those two little nit picks the tool is pretty cool! However I don't think I will be using it myself due to the second point.
Here is a good video describing how this might work. Near the end he shows that even printing out an image that has been "cloaked" and viewing it from different angles can still fool a neural network classifier.
This is awesome and really promising. But the fundamental fact about machine learning is its supposed to approximate/model any scenario so the basic premise of ML would defeat this isn't it? Its a matter of time before someone creates a face recognition model to defeat this.
Finally a comment that addresses how the cloaking works. All the other comments I've seen here wonder how pixel-level changes can prevent recognition. Well, shifting an eye a few millimeters changes the whole face!
They cloaked versions look like different people to me (except for the last). I’m more surprised by nobody mentioning this! They look like relatives, but not the same person.
This might work today, but it won’t work tomorrow.
This is just one side of a GAN, on the next iteration, it will be defeated.
Bottom line is that if a human can recognize, then it is possible for a machine as well.
Also, given that the big networks can just keep throwing more resources at it (I.e. GPT-3), it’s just a matter of increasing the network size to improve feature redundancy.
More accurately, if a human can accurately label inputs and measure outputs, it's possible for a machine. The human eye isn't the peak, just our current standard.
I agree that it won't work tomorrow. To have a system that would continually work you would need to get access to an API that performs facial recognition and then continuously have the system perform queries on that system that would monitor that the facial recognition would fail.
The more people start using this service, the better the AI will learn the differences between real and cloaked images. So eventually anybody can run an unaltered photo of yours through the cloak and it will match up.
Once this technique gets enough attention, a detector for it will be built. Even if the face cannot be recognized, a profile with such picture may be flagged for more scrutiny. This reminds me of using TOR that hides what you visit, and yet likely puts you into a watchlist for surveillance.
I think that a simpler and more robust strategy to achieve good privacy is avoid posting personal information online and social media altogether.
"Hey Bob? Check out this lykahb person. There's something _off_ about them. No Facebook, no Twitter, not even LinkedIn. Probably up to something, we should keep an eye on them. Add them to the list." -- some NSA/GRU/MSS/Mossad contractor
What you're saying points out that there are larger problems of government policy and capitalistic abuse that this software cannot solve.
However, simply not going on the Internet does not solve the problem people care about. People's desired solution is to use the Internet in a personal way and be safe - not just to be safe.
I had a similar idea for a system to add noise to videos to prevent them from being flagged by state censorship systems. Keeping videos of abuse from being deleted from public view in cases like the Great Firewall, for example. I don't have the expertise for implementation yet but I'm glad steps are being made in this direction.
Some are some aren't, there is a vast vast array of different methods, many are not publicly disclosed so I highly doubt the effectiveness of most of these studies.
This may stop some internet marketers, but don't expect it to be effective against large corps and governments.
I wonder if this would be defeated by running an image I wanted to match through it first. Would current state of the art facial recognition match the two cloaked images, or did they already consider that as an attack surface?
Unfortunately these days it is really difficult, borderline impossible to control what images of you are uploaded to the internet. This is discussed in the "Real World Limitations" section of the paper. Even assuming you have no identifying photos online, non-public photos are still analyzed by big companies like Google, Facebook, and Apple, who have access to them through their cloud services (e.g. photos you, your friends or family sync with Google Photos, Apple Cloud). Having just one image correlate to your identification details and you lose anonymity.
121 comments
[ 3.3 ms ] story [ 164 ms ] thread[1]: https://news.ycombinator.com/item?id=23845760 [2]: https://news.ycombinator.com/item?id=23842016 [3]: https://news.ycombinator.com/item?id=23837565
I personally would like to see tests done on facebook by uploading these images and checking if it can recognize it.
p.s. Mobile friendly copy-paste: "Even image compression cannot defeat our cloak. We use progressive JPEG [57], reportedly used by Facebook and Twitter, to compress the images in our dataset. The image quality, as standard by Independent JPEG Group [1], ranges from 5 to 95 (lower value = higher compression). As shown in Figure 15, image compression decreases the protection success rate, but more significantly degrades normal classification accuracy."
> However,we find that none of these transformations defeat our cloaks. The protection success rate remains 100% even when data augmentation is applied to cloaked images5. Ap-plying Gaussian blurring degrades normal accuracy by up to18% (as kernel size increases) while cloak protection success rate remains>98% (see Figure13). Adding Gaussian noise to images merely disrupts normal classification accuracy –the cloak protection success rate remains above 100% as the standard deviation of the noise distribution increases(seeFigure14). Even image compression cannot defeat our cloak.We use progressive JPEG [57], reportedly used by Facebookand Twitter, to compress the images in our dataset. The im-age quality, as standard by Independent JPEG Group [1],ranges from 5 to 95 (lower value = higher compression). As shown in Figure15, image compression decreases the pro-tection success rate, but more significantly degrades normal classification accuracy.
Short answer: No not really. Long answer: Look at the FAQs :)
If the procedure fails then the GAN will know this and it would change the output so that it would pass both outcomes.
I think it does things like subtly change the shape of your eyes and whatnot. That makes people consistently difficult to pick out of a large set of unmodified faces, but a hu man still finds the images recognizable because they're comparing it to a smaller set.
They didn't create the model by training it on the facial recognizers they tested against, and they tested with several different feature extractors.
i love you thank you.
Ben (on behalf of the team)
If a human's neural network can correctly correlate the before/after examples, so can a computer's. They might have found an issue with some modern implementations of facial recognition, sure. But it's a false sense of security to claim "when someone tries to identify you using an unaltered image of you [...] they will fail."
Someone, somewhere said "huh...", and placed another filter into the pipeline to handle these types of images.
While looking for something to sound smart wrt the tank-training myth, I found this interesting page: https://www.gwern.net/Tanks "The Neural Net Tank Urban Legend"
And interestingly, looking at the link for "superresolution needing learned downscalers" found this: https://arxiv.org/abs/1907.12904 "Learned Image Downscaling for Upscaling using Content Adaptive Resampler", code available at https://github.com/sunwj/CAR
So, IDK, seems like this Fawkes approach will be an interesting paper.
color _me_ skeptical, but this is like saying we have functioning AGI; that artificial NNs are the same as the ones we have in our skulls. This to me, is an effect of the over-anthropomorphization of machine learning. It's a bad intuition to have.
However, I do agree. This is just one step in an arms race, and one iteration from being worthless.
https://arxiv.org/pdf/1707.07397.pdf
Neural Networks do not learn what humans learn. They can learn completely different and sometimes much smaller features.
Q: Can't you just apply some filter, or compression, or blurring algorithm, or add some noise to the image to destroy image cloaks?
A: As counterintuitive as this may be, the high level answer is no simple tools work to destroy the perturbation that form image cloaks. To make sense of this, it helps to first understand that Fawkes does not use high-intensity pixels, or rely on bright patterns to distort the classification value of the image in the feature space. It is a precisely computed combination of a number of pixels that do not easily stand out, that produce the distortion in the feature space. If you're interested in seeing some details, we encourage you to take a look at the technical paper (also linked above). In it we present detailed experimental results showing how robust Fawkes is to things like image compression and distortion/noise injection. The quick takeaway is that as you increase the magnitude of these noisy disruptions to the image, protection of image cloaking does fall, but slower than normal image classification accuracy. Translated: Yes, it is possible to add noise and distortions at a high enough level to distort image cloaks. But such distortions will hurt normal classification far more and faster. By the time a distortion is large enough to break cloaking, it has already broken normal image classification and made the image useless for facial recognition.
First, a source image at an approximate resolution that you might find on a social networking site: https://imgur.com/a/9szcC1m
Text output of the tool, which ran for about 3 minutes: https://imgur.com/a/fZtfrmm
The resulting cloaked image: https://imgur.com/a/OSHXdbO
I applied a difference filter between the two images in Photoshop, to show an example of the actual pertubations performed: https://imgur.com/a/q4zC7Ms
Since it's hard to see, I compressed the output to highlight what the program actually changed. It does seem like there is a good amount of disturbance to the image: https://imgur.com/a/1Sx68o3
Now, the real test. First, a Google reverse image search for the original file - identification is pretty bang-on: https://imgur.com/a/5HJwjPx
A Google reverse image search for the cloaked file: https://imgur.com/a/QByXBfS
The only difference I'm seeing is a few images that are one or two images swapped in the "visually similar images" category.
So, I figured that that's the "best case" for the cloaked image - giving the search algorithm the full, unfiltered data, and the program still failed to disguise it. For fun, I thought I would use a "low-pass filter" (Google Lens pointed at my computer screen) as well, just for thoroughness. And the result surprised me!
Here's Google Lens pointed at my screen with the original image open: https://imgur.com/a/1BVRFG0
And here's Google Lens pointed at my screen with the cloaked image open: https://imgur.com/a/uoppuit
So, it would seem that the algorithm's distortions more effectively come through in worse quality images! But, based on my full-resolution result, I wouldn't trust it to disguise something that is being directly uploaded to a social network.
Now, one important note is that reverse image search is probably not using a facial recognition model, but more like image chunk hashing - although I would also consider that something a privacy tool should defend against, which is why I included it.
All in all, very interesting and thanks for convincing me that I should actually test it out.
Here's the original image: https://imgur.com/a/Td4rhoy
And the cloaked: https://imgur.com/a/cPCiCZo
These were both saved as JPG with compression level 8/12. I searched for the cloaked crop (96.jpg) first this time: https://imgur.com/a/FSehQWO
And the original crop (10.jpg) next: https://imgur.com/a/yx4jF0B
This time, Google reverse image search did better at identifying the name of the singer in the cloaked image, instead of just giving the band name for the uncloaked.
Not super scientific since we don't really know what's going on behind-the-scenes with Google reverse image search, but it's certainly one adversary that doesn't seem to be easily fooled if there are other images of "you" out there for it to find. I also tried these small crops in Google Lens with less success (I got unrelated portraits for both images, cloaked or not).
Surveillance software that purports to accurately identify a person across multiple images is not just looking for the same content with some visually insignificant modifications. It's reading your facial structure, attaching your name to it, and searching for it in every image received. Fawkes is working to defeat that specific use case, not all fuzzy matchers in general.
P.S. If you have a human assailant running a reverse image search for photos of you, I think you're well past the point that something like this could be expected to help.
However, if someone is willing to go to that level of effort, the target probably needs to aim for something a little more forceful than tricking Facebook's autotagger.
If they were promising that cloaking would work well on GIS, that'd be a different matter. I can imagine wanting your images to not show up on GIS (because people would use them to try and find the source image on your profile, or something) but it's a different set of constraints at that point.
For cloaking a big use case would be "I took a selfie with a friend and want to share it on my instagram" and your goal is for that instagram selfie to not automatically connect with, for example, a surveillance photo of you at a protest. GIS is obviously not relevant to that scenario.
"similar image" search generally uses "perceptual hashing" which is not related to facial recognition, and not really a privacy risk.
You're not testing against what this tool was designed to defeat :-/
I echo the sentiment of other posters regarding reverse-image search. The original image should not be available to match against. That would be operator failure.
I guess that face rec software will quickly adapt though. That said, we have invisible watermarks that are very resistant to compression or other filters.
That said, given that for most people the threat model is social or work rather than legal, something like this would be terrific to build into consumer insta-photo devices.
Certainly another tool in the privacy toolkit if you absolutely must surrender your likeness to someone else’s computer, but worth bearing in mind that this does not provide (and doesn’t purport to provide) the kind of privacy that strong encryption (or better yet, absent data) can provide.
Edited to add: it’s still damn cool.
Technical solutions have never solved this sort of societal problem. Expecting a few individuals to fight against massive institutions with a little clever math is not going to work.
That rant aside I am curious if this technique will lead to more resilent facial recognition and image parsing techniques to find the shape. Obviously the fact humans can still recognize it is a hint there is some other algorithim possible.
I believe, Model trained on cloaked images would defeat its purpose and make this technique useless.
[0] Su, Jiawei, Danilo Vasconcellos Vargas, and Kouichi Sakurai. "One pixel attack for fooling deep neural networks." IEEE Transactions on Evolutionary Computation 23.5 (2019): 828-841.
[1] Guo, Chuan, et al. "Countering adversarial images using input transformations." arXiv preprint arXiv:1711.00117 (2017).
[2] Liu, Yanpei, et al. "Delving into transferable adversarial examples and black-box attacks." arXiv preprint arXiv:1611.02770 (2016).
(If I were these researchers I'd totally be reaching out to AWS/Azure/GCE for additional research funding... <smirk>)
(So now you just need to somehow get as many cloaked photos of yourself uploaded and tagged to FB as they've collected in the last decade or so...)
I wonder how this holds up when someone takes a photo of that 'protected image'. I can imagine that if these miniscule pixel-scaled changes aren't visible to the naked eye, my crappy 6 megapixel camera will overlook it as well. If I then proceed to feed that image into my image recognition algorithm, is it still protected?
So if your crappy 6 megapixel camera cannot take a clear shot of the cloaked pixels - or effectively applying a blur filter - would also affect the AI detection.
There's quite a lot of comments here that stink of Dunning Kruger candidates, who read the headline and first paragraph, then just started typing their random "wisdom" assuming they're smarter and better informed that the team of PHD researchers who wrote the paper being discussed. (Am I just overly grumpy and judgemental today? Was HN always this bad?)
One thing that I took notice of was how long the program ran on my computer. It took about 5 and a half minutes to obfuscate 4 images on an i9-9900K with the cpu was pegged at 100% the entire time. I can't imagine how long this would take on a low end laptop: especially if I needed to cloak a lot of images in bulk.
Another thing I noticed is that the discoloration that is applied to the final images can be easily mistaken for bruising. If I were to see someones post on social media and they looked like my results I'd be inclined to think that the poster is recovering from a bad fight or is a victim of abuse.
Other than those two little nit picks the tool is pretty cool! However I don't think I will be using it myself due to the second point.
https://www.youtube.com/watch?v=4rFOkpI0Lcg
In practice, building new training sets is much more expensive than slightly changing the cloaking algorithm.
This is just one side of a GAN, on the next iteration, it will be defeated.
Bottom line is that if a human can recognize, then it is possible for a machine as well.
Also, given that the big networks can just keep throwing more resources at it (I.e. GPT-3), it’s just a matter of increasing the network size to improve feature redundancy.
Adversarial examples transfer between different models trained on different datasets with different architectures.
A new model from yesterday's data is essentially the same architecture, just with some fluctuations in decision boundaries.
Might it affect the success over time? Sure. But not tomorrow.
I think that a simpler and more robust strategy to achieve good privacy is avoid posting personal information online and social media altogether.
However, simply not going on the Internet does not solve the problem people care about. People's desired solution is to use the Internet in a personal way and be safe - not just to be safe.
This may stop some internet marketers, but don't expect it to be effective against large corps and governments.
In fact, it's basically the entire problem.