I would love to see a version of this applied to both VR and AR but it seems that the current implementation is still slow for the performance that VR requires.
From the site:
"During inference, our system operates at 25fps on 320x240 images and 4-5fps on 800x1100 images using a GTX1080 graphics card."
You don't necessarily need better FPS from the pose tracking, because you can use optical flow to interpolate and predict movement. These sort of techniques are already commonly used for VR positional tracking off camera input.
That's exactly what I had in mind, but I'm no expert. And just wait until they've got basic, fluid object recognition down. I've already seen some impressive, old work on coherent in-painting of images.
It seems like all these deep learning discoveries are relatively composable too.
If you thought face swapping was bad, just wait until someone uses this tech to create AR goggles that let the wearer see everyone “naked”, like having Superman X-ray vision
I think that's a really good question. IMO this is an essential part of any pipeline aiming to do that, but there's a bunch of hurdles with neural networks that's haven't been crossed yet. The first one that comes to mind is that I'm not aware of any work which would try to teach neural networks to imagine what someone would look like from different angles (aside from relatively restricted settings like facial frontalization). Second, you talk of videos. That means the person needs to look 'consistent' from frame to frame. This isn't a solved problem either, especially when it comes to GANs. I'm sure there are a few other areas that need to be solved.
17 comments
[ 0.31 ms ] story [ 45.2 ms ] threadFrom the site: "During inference, our system operates at 25fps on 320x240 images and 4-5fps on 800x1100 images using a GTX1080 graphics card."
For good VR/AR experiences you'll want better FPS
It seems like all these deep learning discoveries are relatively composable too.