Googling "nvidia face generator" lead me to "A Style-Based Generator Architecture for Generative Adversarial Networks," (6 Feb 2019) a paper showing the faces on the site.
GAN creating faces purely at the pixel level still seems a strange approach to me. In some years it will feel very restrictive. I guess it's the only tractable method at the moment.
Is anyone working on a GAN to generate bone structure then flesh and skin/mouth/eyes textures and pipe the result in a ray tracer?
It's incredible what can be done in 2D solving directly for the result, but imagine where this goes when this works in volume and multiple levels more driven by physics.
Upon further reflection, my observation's mostly a straw man. HIPAA covers a patient's face when a dermatologist takes an image just as a x-ray covers a bone structure.
Privacy is hard to talk and reason about without defining everything specifically.
This projects images were sourced from Flickr. You can find medical imagery on Flickr reasonably easy as well it turns out.
That defeats the purpose. The purpose of deep learning is to let the machine solve the problem using only goal data. If you are going to program it with bones and flesh models then you don't need a neural net.
I disagree. If the purpose is simply to get a single-use photograph, then yes. But for this type of application I think we want to create a virtual human that can, at the very least, be photographed from two different angles and ideally put into motion. To create an virtual entity.
You wouldn't "program" it with flesh and bones, you would generate a life-like but original new skeleton in the same way we generate these images, except the space of solution is the space of possible skeletons instead of possible pixel configurations. And then generate soft tissues that are also original, conforming to biology constraints and also constrained by the underlying skeleton. Same for skin, created from scratch but believable and driven by the underlying tissue.
With enough training data, I would think you could create alternate views from the first view via ML methods, rather than doing skeletal structure by ML and then physics modeling to get views.
But what if you can train other feature lines that will generate backgrounds and other humans? Then add in another feature line that can morph an existing human line into different views, poses and/or animations.
Yes. You can take a particular latent point and generate slight variants of it. You could also apply different style/noises to change the overall appearance like orientation. Take a look at the StyleGAN paper's figures, the StyleGAN video, or the various interpolation videos people have made.
The presentation is very interesting. That's always what amazes my with those GAN outputs. These people do not actually exist. Obviously, there are some funky examples. Nothing wrong with mine at first, although his buddy should probably see a doctor: https://imgur.com/a/dkS8Ux5
With tens of thousands of training images, and hundreds of dimensions in the latent space, I don't think I would assume this is true. You may be thinking one feature is "eyes", but the eyes may instead be built out of 10 sub-features. Those sub-features may be inextricably linked to other, identifiable, macro-features.
There was a study done on faces and beauty. They created faces based on global averages of features and found these (synthetically invented) faces to rank even more highly on the beauty scale.
A painting is usually immediately recognizable as a work of art / fiction. Do you want to appear as a team member on an escort service site which uses automatically generated placeholder images to protect their employees? Do you want your face on a billboard ad for Viagra?
There is a huge difference between "hey, that painting looks like you" and "hey, that is a photograph of you".
That almost certainly isn't needed for a project like this, but I wonder if it would be needed to try to make money off this. It might be like the whole "The events depicted in this movie are fictitious..." warning you see at the end of almost every movie regardless of it is some realistic and plausible story or if it something like Avatar.
Since it uses a deep neural network, I don't think it's "thousands a second". Also, you can download an image and crop a face out of it in several seconds. You could even automate the process.
The biggest problem is transferring faces to existing photos. It was hard to do manually. Now it's much easier. Also, people are generally trained to ignore various artifacts by CGI-ridden movies and compression algorithms. So much of our notion of how the world looks comes from digital imagery, it's kind of scary.
I think we need to change the threshold of quality for an image/video to constitute "proof" of any kind. You can hide most of the weird artifacts by scaling things down or passing them through heavy compression.
> Since it uses a deep neural network, I don't think it's "thousands a second".
The generator is like 150MB. The forward pass is <0.1s. Hypothetically you should be able to generate on a decent GPU like a 1080ti with 11GB VRAM at full utilization <730 images per second. Use a few GPUs and you're at thousands per second.
Where is this data from? I see 300MB model on their Google Drive. And if I understand correctly, you also need source and destination images to transfer styles from and to, so it's not like the model generates photos out of thin air.
The 300MB model covers both the G and D. You only need G to generate. The style transfer is just noise. And I time my own 512px anime StyleGANs at ~21 images per second per model; half that throughput to account for the increased model size and depth of a 1024px. No matter how you tweak the numbers - halve it again if you wish! - it's clear that thousands per second is entirely attainable with a few GPUs at low cost. (For comparison, 8 V100s is ~$7/hr on AWS; 10x1080ti is ~$1.3/hr on Vast.ai.)
Stolen ones are easier to do reverse image detection on and expose as fraudulent. I see this a lot on Twitter - bots pull a mix of stolen profile pics, bios, etc.
I'm seeing some characteristic artifacts in most of these pictures. A hair halo floating just outside of the head is pretty common, and there's a sort of rainbow fringing that was very common in the deep dream postings that I'm still seeing popup...
... but all, or almost all, of these would be irrelevant at a profile pic size. At that size, assuming these aren't just recapitulations of the training data (and I assume they aren't) this technique appears to be 99%+ successful.
Also, look at the non-face stuff. Some backgrounds are just "incredibly blurred vaguely landscapy stuff", which is plenty realistic, but I've seen the algorithm attempt wood grain, which went poorly. I've seem some bizarre patchwork backgrounds, and one picture had a person cut off to the right like a single photo trimmed from a family photo, and the cut-off person was some sort of SCP-monstrosity mercifully cut off by the edge of the photo. Still, the success is impressive. The failures are definitely going from "in your face" to "easy to ignore/miss".
Fix up the training data a bit and this'd be a profile pic machine.
Every one of these pictures has something 'off' with them. But it takes a while to notice. My first three - the teeth were odd, there were small, regular rectangles out of the teeth. Then it was the eyes. Specifically the iris was the wrong shape and had too much glare from what I assume is a camera.
Every single picture, if you really look at it, is disturbing for reasons you can't pick out. It's definitely hitting the uncanny valley. It's juuuust human enough to blend, but not human enough to avoid the creeping feeling of dread.
But that being said, if I'm cruising forums and see this in a thumbnail size, I'm not going to be able to pick out that it's not a real person.
they're locally plausible but globally everything's slightly wrong: the measurements of features, the mixture between high-resolution and sudden blurriness, the occasional warping effect, the shifting perspective, the eyes don't sit in the skull correctly, the hair seems to be intersecting the forehead sometimes, every area seems to be located in a different space & there's no three-dimensional coherence to it
right off the bat the two photos [in each post] have some sort of artefact on the right temple, gave it away as a manipulated image on first glance. and there are halos and blur.
im also wondering just how far these fakes could go? there is such high resolution with very common cameras now that the reflection of what a person is looking at is visible in the lens of thier eye. [its even a zero day] the AI is going to need someway of creating a fake setting to go with the face, and fake EXIF data that matches the fake camera model that would have taken the picture.
With some of them you notice it straight away like the woman with something sticking out of a hole in her cheek:
https://fb.pics/image/38yjt
and the woman with the mutilated left ear:
https://fb.pics/image/380UC
and the child with the adult eye bags:
https://fb.pics/image/38Jva
One thing it almost always does wrong is glasses.
https://fb.pics/image/38NNu
And apart from pictures of young children, most of them have strange vertical wrinkles under the eyes, even when everything else is relatively convincing.
Facum Ipsum sounds like an entertaining side-project. Coupling one of these images with a random profile generator for each employee and outputting to JSON. One button press would populate your app with relevant data.
What's funny is "Like Lorem Ipsum, but for people" is the tagline for randomuser.me, which currently uses stock photos with random user data. Combine these 2 and you got yourself a party.
Run it several billion times, create an Earth-sized social network, then give it content with a meme generation system like Dank Learning https://arxiv.org/abs/1806.04510
The first 3 I viewed were all incredible except for a weird little flaw around the edge of the hair/background line, first 2 had an unnatural notch out of their hair and the third was a weird discolouration/thinning of hair.
The first time I heard of GAN's creating faces of people who never existed was from the show "Person of interest", where the "machine"(an AI) creates a face and assigns it to its identity.
biggest weakness of this system seems to be generating realistic backgrounds, the faces look amazing - but around the edges some photos appear to "swirl" with the background
I've never read any benchmarks around render time on sophisticated GANs, so maybe the answer to this is obvious, but: Is this showing a random selection from a set of offline-generated images, or is the GAN actually generating these on each request?
This is what always bothers me about demonstrations of the technology. There's almost always a extra hidden layer of human curation of the output so we only see the examples that are most interesting (90% amazing result with a mixture of hilarious/horrible bad results for flavor)
This work is impressive, I don't mean to take anything away from it, but if the author had to filter through 1000 images to select the 5 I saw, that's ... disappointing?
It's worth noting that although at a first glance the face looks extremely realistic, there are some details that don't quite make sense and hint at a randomly-generated face.
- weird hair above the person's right eye, that doesn't match with the overall hairstyle (the patch of short hair) or realistic hair behaviour (straight bit of hair)
- what seems like beard on the chin, with unrealistic lighting
- hair turns into leaves at the bottom
- weird reflex in the left glass
- mismatching shapes for glasses (there's a small bump only on the right glass)
I noticed that there's generally odd texturing and hair placement. Wrinkles on an otherwise smooth face, or in weird places / directions. Hair of a mismatched color. The unusual facial texturing seems to occur more on the right side of these images.
It's also worth noting that a couple of years ago, most GAN-generated faces looked obviously wrong at 128x128px. It's entirely plausible that this approach is ultimately a dead end, but it's also plausible that we're at a crucial inflection point in the development of computing.
Hrm, your example was quite glaring in its flaws, most of the images I saw, on the other hand, looked quite flawless. I actually came here to disagree with someone else who stated the images he saw had alien-like alarming characteristics, or something along those lines. I can’t tell most of these are fake even at full size on an XS Max.
> Recently a talented group of researchers at Nvidia released the current state of the art generative adversarial network, StyleGAN, over at https://github.com/NVlabs/stylegan
> I have decided to dig into my own pockets and raise some public awareness for this technology.
> Faces are most salient to our cognition, so I've decided to put that specific pretrained model up. Their research group have also included pretrained models for cats, cars, and bedrooms in their repository that you can immediately use.
> Each time you refresh the site, the network will generate a new facial image from scratch from a 512 dimensional vector.
247 comments
[ 4.6 ms ] story [ 164 ms ] threadhttps://arxiv.org/pdf/1812.04948.pdf
Is anyone working on a GAN to generate bone structure then flesh and skin/mouth/eyes textures and pipe the result in a ray tracer?
It's incredible what can be done in 2D solving directly for the result, but imagine where this goes when this works in volume and multiple levels more driven by physics.
Of course, this is also why training data is more difficult to acquire, as you mention.
Privacy is hard to talk and reason about without defining everything specifically.
This projects images were sourced from Flickr. You can find medical imagery on Flickr reasonably easy as well it turns out.
You wouldn't "program" it with flesh and bones, you would generate a life-like but original new skeleton in the same way we generate these images, except the space of solution is the space of possible skeletons instead of possible pixel configurations. And then generate soft tissues that are also original, conforming to biology constraints and also constrained by the underlying skeleton. Same for skin, created from scratch but believable and driven by the underlying tissue.
https://www.youtube.com/watch?v=HO1LYJb818Q
Don't underestimate how hard it is to get photos of people for scams.
And they are very important for building trust.
This should be big in the blackhat world, it's possibly the start of whole new scams against a whole new set of people.
https://imgur.com/a/ptgKbh1
https://imgur.com/a/2iA7t3N
But seriously: it may be essential for legal reasons to be 100% sure that an automatically generated face does in fact not depict a real-life person.
There is a huge difference between "hey, that painting looks like you" and "hey, that is a photograph of you".
Previously it wasn't trivial to do a GAN image generator, now as this site shows it's, if not trivial, also not particularly hard.
The biggest problem is transferring faces to existing photos. It was hard to do manually. Now it's much easier. Also, people are generally trained to ignore various artifacts by CGI-ridden movies and compression algorithms. So much of our notion of how the world looks comes from digital imagery, it's kind of scary.
I think we need to change the threshold of quality for an image/video to constitute "proof" of any kind. You can hide most of the weird artifacts by scaling things down or passing them through heavy compression.
The generator is like 150MB. The forward pass is <0.1s. Hypothetically you should be able to generate on a decent GPU like a 1080ti with 11GB VRAM at full utilization <730 images per second. Use a few GPUs and you're at thousands per second.
... but all, or almost all, of these would be irrelevant at a profile pic size. At that size, assuming these aren't just recapitulations of the training data (and I assume they aren't) this technique appears to be 99%+ successful.
Also, look at the non-face stuff. Some backgrounds are just "incredibly blurred vaguely landscapy stuff", which is plenty realistic, but I've seen the algorithm attempt wood grain, which went poorly. I've seem some bizarre patchwork backgrounds, and one picture had a person cut off to the right like a single photo trimmed from a family photo, and the cut-off person was some sort of SCP-monstrosity mercifully cut off by the edge of the photo. Still, the success is impressive. The failures are definitely going from "in your face" to "easy to ignore/miss".
Fix up the training data a bit and this'd be a profile pic machine.
Every single picture, if you really look at it, is disturbing for reasons you can't pick out. It's definitely hitting the uncanny valley. It's juuuust human enough to blend, but not human enough to avoid the creeping feeling of dread.
But that being said, if I'm cruising forums and see this in a thumbnail size, I'm not going to be able to pick out that it's not a real person.
I'd say you're correct that it is still in it, but it is clearly climbing up the other side now. We're past the minima of verisimilitude now.
https://thenewstack.io/deep-learning-ai-generates-realistic-...
Run it several billion times, create an Earth-sized social network, then give it content with a meme generation system like Dank Learning https://arxiv.org/abs/1806.04510
This work is impressive, I don't mean to take anything away from it, but if the author had to filter through 1000 images to select the 5 I saw, that's ... disappointing?
This is what I got: https://i.imgur.com/iCfzjkZ.jpg
In no specific order:
- weird hair above the person's right eye, that doesn't match with the overall hairstyle (the patch of short hair) or realistic hair behaviour (straight bit of hair)
- what seems like beard on the chin, with unrealistic lighting
- hair turns into leaves at the bottom
- weird reflex in the left glass
- mismatching shapes for glasses (there's a small bump only on the right glass)
Also a LOT more in this subreddit: https://old.reddit.com/r/SyntheticNightmares/
Clearly, this algorithm has captured a dryad.
> Recently a talented group of researchers at Nvidia released the current state of the art generative adversarial network, StyleGAN, over at https://github.com/NVlabs/stylegan
> I have decided to dig into my own pockets and raise some public awareness for this technology.
> Faces are most salient to our cognition, so I've decided to put that specific pretrained model up. Their research group have also included pretrained models for cats, cars, and bedrooms in their repository that you can immediately use.
> Each time you refresh the site, the network will generate a new facial image from scratch from a 512 dimensional vector.
(https://www.facebook.com/groups/DeepNetGroup/permalink/80536...)