I was looking for even better quizes TBH because I have played and seen images by some new tools (Ideogram, Modjourney etc) and for some images (shared on their respective community networks) you have to look very very closely to spot issues.
Found a few more quizes here. See if you get full score here too.
The NBC quiz was a bit silly - "would Degas have used such loud blues?" when one ballerina's arm has melted into another one's shoulder. And the "wrong hair texture" for the woman on the right in #20 could just as well be a stylistic choice rather than indicating an ML-generated image.
8/10 here with no such experience and just clicking on instinct without trying to analyze the picture. The rocket launch and Mt Fuji are the two I missed. I think I would've also gotten at least the rocket launch had I bothered to zoom in (did this on mobile).
I got 10/10 and I know exactly zero about VFX and CGI. I did it really fast too. The only one that I hesitated was the aurora borealis.
That said, this isn't some sort of great victory. I expect that soon my intuition won't be good enough. Telling AI from real will never get easier, it will only ever get harder.
I find it interesting how often I identified the difference using a different detail than the article points out. AI Hong Kong has four copies of the same building. The aurora is supposed to be at night but has sunlight on the mountains.
These were pretty terrible. 10/10 if you've literally ever seen an AI generated image before. Totally not representative of the state of the art. In 5 years I doubt anyone but experts can tell by the naked eye in most cases.
10/10 without checking any details in the image. I'm a photographer tho. Those AI files were pretty easy to point out as they all have this "glow" or softness/playdough aspect especially on skins that real images don't have. Also, that helps that a lot of those real pictures were artificially sharpened, something an AI software probably won't care about.
9/10 purely going by which one didn't look "better" (hieroglyphs were the miss and even then I was 50/50 on that). Also the "real" picture of kids doing an art project is clearly not real (i.e. everyone involved is doing a pose for a stock photo)
For everyone complaining about how easy it is and how you got 10/10: You are not the target victims. These don’t fool us because we have watched AI images from their birth to now and know exactly what to look for. These will easily fool me my elderly parents and their entire church group. Go ahead and send this quiz to anyone not steeped in Internet technology and observe their results.
More than find people who can be duped I wonder how effective this kind of demo really is at its goal. My gut feeling is it's mildly effective for low effort fakes in the current moment and probably nearly completely useless for images generated a year from now. Like spam I think the only practical answer will be catching 98% via automated methods and then convincing people not to do anything too rash/important based on being shown unsolicited digital content.
10/10, but I find the explanations to be somewhat lacking. "Not real hieroglyphics". Yeah, of course, should have paid better attention in my classes on old Egyptian writing I guess? I feel a proper spotting technique should be generic, not specific to the subject matter at hand which you might or might not know.
For example, in the Egyptian example, I thought the outlines of peoples feet and legs to be too precise and devoid of the little imperfections of human drawings.
I got the egyptian one right (all of them actually) and I was very confident of it despite not really knowing anything about heiroglyphics. The technique was simply, which one is more stereotyped? The AI one just has the classic figures and "heiroglyphs" and that's it. The real one has tons of detail - odd plants, a banquet, stacked pots, a king with a face that has personality. I look at it and am drawn into wanting explanations for the things in it. It's clearly the work on someone trying to tell a story, rather than trying to "look egyptian". The same basic idea worked for basically all the others - which photo has more odd, incidental, imperfect detail? The AI ones are "generic" in the same way ChatGPT prose is.
10/10 but a few years from now, I might do no better than chance. I was amazed in the Mount Fuji image that the reflected clouds were correct based on the ones it drew in the sky. Those details are on opposite sides of the image!
I was hoping these were going to be actually good AI images, not those stereotypical cases where they screw up hands, smudge details, have cartoonish hyper-saturation/hyper-sharpness, have nonsense text, etc. This isn't really an interesting test.
Side note: I doubt they are able to verify that none of the images were edited by a human. E.g. the Mt Fuji one is from Adobe Stock, so I wouldn't be surprised at all if it had reflections corrected by a human.
AI images don't have any noise, these were the easiest photos. The one with the Titanic was too obvious: how could they make a great colorized photo back then? I was wrong with the close up photos of the animals though.
There are very few tricks I rely on:
1. Skin is to smooth and shiny
2. Some kind of blur or glow
3. clear logical mistake in the picture
4. I know the picture/artwork already
5. AI could not have come up with something this deep.
6. Hands
However there are several factors working against you:
1. Real photographers tend to like the moody/glowy/shiny look of pictures.
2. Your knowledge about the topic comes from the internet and above mentioned photographers. You trained on the same dataset the AI did.
3. In these tests you know one is real and one is AI
4. In real settings the person publishing the picture can avoid do the quality control to make sure it isn't obvious, or collect data from tests like these to improve first. (Next level of captcha: is this image real or fake?)
7/10, would have been 8/10 if I'd done them all the same way, which is just a gut reaction to whichever I immediately think is AI.
I briefly let some detail convince me to switch my vote on the first one (frog) and got it wrong, and from there decided I either needed to spend time with each image or go with my immediate gut belief. Gut is pretty good on these; spending the time to analyze probably even better, but spending just enough time to notice a detail or two will probably push accuracy to 50%.
I don't really care that I got the aurora borealis wrong, those typically look unreal anyway, are usually long exposures, I have so little experience with the subject material.
I also got the pangolin wrong, which was the only one aside from borealis where I had no strong gut feeling. I don't have an understanding why this
one was harder for me.
There's no way to run the experiment twice but if someone made a larger set I'd be curious to find out if I do better or worse on a desktop than on my phone (how I did these), I find my experience of images is extremely different between the two.
I'm worried that this form of education will enforce a false sense of security as AI generated images improve.
It's simply an inaccuracy in being able to perfectly settle on the hyperspace between information learnt so far / or even a lack of fidelity with current state of the art.
As time goes on, what we're all learning and becoming great at detecting will adversarially learn to improve. The interesting part is it's inevitable, every step towards the goal regardless of who it's by will achieve the same result.
Consider the current AI race with providers trying to keep their tech. out of public reach due to "safety concerns". If it can be thought about, it will be achieved by open source, and if it can be output by any system, it will be used as training data for the rest of them.
I'm surprised they didn't point out the extra toes in the AI image and the other hind-leg anatomy that looks weird. Though the biggest tell to me was the more static and stereotypical pose.
> Students Doing Art Projects
This is another more A/B comparison thing, but it's striking how static the AI images is, and fails to simulate the attention and emotion of real people.
> Hong Kong at Night
I don't think noting inconsistencies with the actual scene are helpful for people to actually spot an image like that.
> Ancient Egyptian Papyrus
Ditto with knowing what "accurate hieroglyphs" are.
10/10, but knowing one of them is AI and the other isn’t makes it so much easier. Here you can just pick the one that has more detail that doesn’t really add to the aesthetic. In the real world the question is do you think this image is AI or not?
I got 0 out of 10, turns out they wanted me to select the correct photo and not select the fake. How was this obvious to everyone else, certainly wasn't to me
38 comments
[ 2.1 ms ] story [ 64.0 ms ] threadFound a few more quizes here. See if you get full score here too.
https://www.nbcnews.com/specials/ai-generated-art-photo-quiz...
https://real-or-fake-the-ai-game.onrender.com/
The more you look, the harder it gets. Just go with your intuition. If it looks like a Unity game engine screenshot, it's fake.
That said, this isn't some sort of great victory. I expect that soon my intuition won't be good enough. Telling AI from real will never get easier, it will only ever get harder.
all the rest correct
Side note: I doubt they are able to verify that none of the images were edited by a human. E.g. the Mt Fuji one is from Adobe Stock, so I wouldn't be surprised at all if it had reflections corrected by a human.
Also reminded me of this Rainbolt (Geoguesser guy) video: https://www.youtube.com/watch?v=lxFlpDq26hM
There are very few tricks I rely on: 1. Skin is to smooth and shiny 2. Some kind of blur or glow 3. clear logical mistake in the picture 4. I know the picture/artwork already 5. AI could not have come up with something this deep. 6. Hands
However there are several factors working against you: 1. Real photographers tend to like the moody/glowy/shiny look of pictures. 2. Your knowledge about the topic comes from the internet and above mentioned photographers. You trained on the same dataset the AI did. 3. In these tests you know one is real and one is AI 4. In real settings the person publishing the picture can avoid do the quality control to make sure it isn't obvious, or collect data from tests like these to improve first. (Next level of captcha: is this image real or fake?)
[0]: https://news.ycombinator.com/item?id=41033341
I briefly let some detail convince me to switch my vote on the first one (frog) and got it wrong, and from there decided I either needed to spend time with each image or go with my immediate gut belief. Gut is pretty good on these; spending the time to analyze probably even better, but spending just enough time to notice a detail or two will probably push accuracy to 50%.
I don't really care that I got the aurora borealis wrong, those typically look unreal anyway, are usually long exposures, I have so little experience with the subject material.
I also got the pangolin wrong, which was the only one aside from borealis where I had no strong gut feeling. I don't have an understanding why this one was harder for me.
There's no way to run the experiment twice but if someone made a larger set I'd be curious to find out if I do better or worse on a desktop than on my phone (how I did these), I find my experience of images is extremely different between the two.
It's simply an inaccuracy in being able to perfectly settle on the hyperspace between information learnt so far / or even a lack of fidelity with current state of the art.
As time goes on, what we're all learning and becoming great at detecting will adversarially learn to improve. The interesting part is it's inevitable, every step towards the goal regardless of who it's by will achieve the same result.
Consider the current AI race with providers trying to keep their tech. out of public reach due to "safety concerns". If it can be thought about, it will be achieved by open source, and if it can be output by any system, it will be used as training data for the rest of them.
I'm surprised they didn't point out the extra toes in the AI image and the other hind-leg anatomy that looks weird. Though the biggest tell to me was the more static and stereotypical pose.
> Students Doing Art Projects
This is another more A/B comparison thing, but it's striking how static the AI images is, and fails to simulate the attention and emotion of real people.
> Hong Kong at Night
I don't think noting inconsistencies with the actual scene are helpful for people to actually spot an image like that.
> Ancient Egyptian Papyrus
Ditto with knowing what "accurate hieroglyphs" are.