To be fair, raw data from "real" stereoscopic approaches looks fairly similar. Definitely a good result, but certainly something I have seen before in a comparable quality. Hard to judge from the examples though.
Obviously it does not literally add 1/10 of a dimension. However, saying "2.1 dimensional" gives people the idea of "oh, it is still mostly the same as a 2 dimensional thing, but is a little bit closer (in some sense) to being 3 dimensional.".
Interestingly, FB folks used a vastly simpler network architecture, but it probably still works because they have access to so much real data. In contrast, the Adobe paper had a fairly involved network with pretrained VGG features, possibly to compensate for only having synthetic data to train on.
It's also worth noting that FB doesn't need a lot of accuracy in the depth data for this to be "good enough" for a simple wobble effect on a phone.
It definitely doesn't need to be held to the same standards that would be held for a monocular depth estimator for, say, an autonomous vehicle or robot; in FB's case it doesn't matter whether the tree is 4m or 6m away, and really only matters that it's roughly behind the dog or cat or whatever.
Separately, depth estimators like this are also useful to fake the depth of field effects of large-aperture lenses, which I'm pretty sure is used on the latest Pixel and iPhone in their background-blurring "Portrait" mode; Facebook could quite possibly include that as an in-app feature to make such effects accessible to everyone with cheaper phones as well.
How many people like watching 3D movies at home or at a movie theater? Sometimes old fashioned 2D images are just fine. The content, story and artistic aspects of the image is what matters, not some cool half-baked 3d trick that is... frankly a distraction.
I don't mean to criticize the research and engineering that went into doing something like this - it truly remarkable. I refer to the application possibilities that we might see in the future - imagine the web full of these 3d images, I think that would be a pretty noisy and distracting place.
If you extrapolate where social media apps are taking us, first it was to "keep in touch" (however superficial) with people, even when it's a tweet that "they're pooping" or an IG of what they're having for lunch. And then came Periscope livestreams and Snapchat stories (stolen by Instagram and Facebook), so you can see a live video of what your friends are doing. FB bought Oculus, I wonder if their fantasy of the future is for realtime immersive sharing of experiences, your friend is on top of the Eiffel tower, you put on your goggles, and boom, you're there with them with a 360 degree video.
At least that's the technological side, personally I think this simulacra of friendship/socializing is terrible.
Anyone remember an iPhone app canned Seene (I think) that did similar about 7 years ago?
It lacked the AI aspect but essentially you took several photos from different angles and it created some form of depth map with the central image as a texture (if I remember correctly)
I was amazed by this at the time.
https://get.display.land is also pretty good. It did a passable job of our local statue[1] in about 3 minutes of capturing + [some time of their cloud processing].
(Now it's not in the same ballpark as the capture I made with 5 minutes of taking ~90 photos and the ~20 hours of Meshroom processing but that's understandable.)
The difference here is that they (FB) are able to create 3D model based on only one photo. If you have multiple photos from different angles, you can just use geometric relationships to figure out where the points lie in 3D space. However, with only one picture, you need to "guess" (using ML), as there's not enough data for proper 3D reconstruction.
>If you’re more interested in how this impacts you, just know this: whether you want to convert “decades-old family photos” to 3D, or you want to convert your professional portraits into 3D-like creations, FB now lets you do that. You just need to upload it using the Facebook app on “an iPhone 7 or higher, or any recent midrange or better Android device.”
...aaaand that's the motivation for this feature - persuading people to upload more photos to Facebook.
lol, you say this like it's a revelation. Why would a company build a feature in their product and not expect people to use their product to use the feature.
There's a significant difference between "this works for all your current pictures on Facebook" and "you need to choose to upload some specific pictures on this specific app to use this feature".
The latter will get you a far better information on what people want to see turned into 3D.
The text is ambiguous, but I would interpret it as "Photos need to be on Facebook to do this, so those not already on[...]" rather than "This works only on Photos newly uploaded".
Why would they need to restrict the feature to new photos only to get that information? They could just use telemetry and see what photos people pick if they wanted to.
Wouldn't there need to be some business motivation to push that all the way through QA and localization and all the other steps before it hits production?
Call me silly but I am extremely certain Facebook is not exactly struggling to attract tech talent to the point that they'd build, test and deploy non-trivial features just to do so.
Or maybe it's being one of the biggest companies in the industry and having hundreds of billions of dollars to throw behind compensation and other perks.
Nah, you're right, they put entire teams to work building and maintaining user-facing features so they can add a couple thousand more applicants to the hundreds of thousands they already get.
I remember playing with effects like this ~8 years ago. My first instinct would be to laugh at the "AI" part, but I'm guessing the point of the exercise isn't the parallax effect - the point is that they've actually managed to get a neural network to do that.
54 comments
[ 3.3 ms ] story [ 121 ms ] threadIt's entertaining at least :)
Parallax effects are not 1/10 of a dimension in any sense of the word.
there also wasn't a Lion King 2.5 until there was.
https://news.ycombinator.com/item?id=20978055
Interestingly, FB folks used a vastly simpler network architecture, but it probably still works because they have access to so much real data. In contrast, the Adobe paper had a fairly involved network with pretrained VGG features, possibly to compensate for only having synthetic data to train on.
It definitely doesn't need to be held to the same standards that would be held for a monocular depth estimator for, say, an autonomous vehicle or robot; in FB's case it doesn't matter whether the tree is 4m or 6m away, and really only matters that it's roughly behind the dog or cat or whatever.
Separately, depth estimators like this are also useful to fake the depth of field effects of large-aperture lenses, which I'm pretty sure is used on the latest Pixel and iPhone in their background-blurring "Portrait" mode; Facebook could quite possibly include that as an in-app feature to make such effects accessible to everyone with cheaper phones as well.
I don't mean to criticize the research and engineering that went into doing something like this - it truly remarkable. I refer to the application possibilities that we might see in the future - imagine the web full of these 3d images, I think that would be a pretty noisy and distracting place.
At least that's the technological side, personally I think this simulacra of friendship/socializing is terrible.
Are we now going to remake all pictures into """3D""" and leave the boring 2D pictures in dust ?
https://en.wikipedia.org/wiki/Photogrammetry
There is a good introduction to production use in this GDC presentation from DICE:
https://www.gdcvault.com/play/1023272/Photogrammetry-and-Sta...
Examples of applications that are usable for individuals include:
https://www.3dflow.net/3df-zephyr-pro-3d-models-from-photos/
https://www.agisoft.com/
https://www.capturingreality.com/
And free/open source/academic:
https://colmap.github.io/
http://ccwu.me/vsfm/
https://alicevision.org/
(Now it's not in the same ballpark as the capture I made with 5 minutes of taking ~90 photos and the ~20 hours of Meshroom processing but that's understandable.)
[1] https://www.display.land/s/4QeZc8gT77beNLx44OdvDT
...aaaand that's the motivation for this feature - persuading people to upload more photos to Facebook.
The latter will get you a far better information on what people want to see turned into 3D.
5 imaginary dollars says it's just something their AI team built and thought was cool so they rolled it out.
Nah, you're right, they put entire teams to work building and maintaining user-facing features so they can add a couple thousand more applicants to the hundreds of thousands they already get.