We built this because we wanted 3D experiences without needing a VR headset.
The approach: use your webcam to track where you're looking, and adjust
the 3D perspective to match.
It creates motion parallax - the same depth cue your brain uses when you
look around a real object. Feels like looking through a window instead of
at a flat screen.
Known limitations:
- Only works for one viewer at a time
- Needs decent lighting
- Currently WebGL only
We're still figuring out where this is genuinely useful vs just a novelty.
Gaming seems promising, also exploring education and product visualization.
I don't know if you designed it for a specific monitor but, feedback. It tried using it on my M1 Mac.
First thing, there is no loading indicator and it takes too long to start so I though it was broken a few times before I realized I had to wait longer.
Second thing, although it was clearly tracking my head and moving the camera it did not make me feel like I'm looking into a window into a 3d scene behind the monitor.
These kinds of demos have been around before. I don't know why some work better than others.
Isn't it because the webcam FOV is unknown, which is needed to estimate distance from pixel (along with face size, but that should be less variant). The three.js demo had the strength parameter that can be used to calibrate. The iphone app is pre-calibrated for the most common devices, I believe.
It was definitely useful and appreciated on the "New" Nintendo 3DS XL, which also used a camera to track your eye movements and adjust the divergence accordingly. I hate the fact that Nintendo abandoned this technology because experiencing Ocarina of Time and Star Fox 64 in 3D was world-changing to me.
I'd say I'm not the only one who misses this technology in games, because a used New 3DS XL costs at least $200 on eBay right now, which is more than what I paid new.
I always thought 3D would combine really nicely with a ray traced graphics full of bright colors and reflections, similar to all those ray tracing demos with dozens of glossy marbles.
Has it though? And what "market" are you referring to here?
Fully agreed that if you want 100% full 6DOF immersion - go and pay hundreds or even thousands of dollars to wear a heavy and cumbersome headset on your head. We're not disputing that or thinking of competing with that.
What we're saying is that there may be a much larger market consisting of people who are not ready to commit to pay so much money to wear something that will give them motion sickness after 10 minutes.
If you're developing a VR game your market consists of 50 million people around the world who owns a VR headset. That's great. But since you already built the VR world in 3D, you could also open up the market to billions of people who want to play your game but on their own devices.
Admittedly, it won't be the same experience, but it could be a "midpoint". Not everyone can afford and is willing to pay for a VR headset.
The 3DS is different, it's using a lenticular screen so your eyes actually see different images! The eye tracking allows it to work even if the position of your eyes changes (ie because you moved your head).
Presumably developers could have combined this with parallax head tracking for an even stronger effect when you move your head (or the console), but as far as I know no one did.
Yeah Nintendo 3DS XL was awesome, but even then, you'd have to use that one specific console in order to be able to play that game.
What we're thinking is to enable this technology - as long as there is a camera and a screen - instantly accessible across billions of devices.
That means that the 3D effect would be applicable not only for games built for that specific console - but for any and all games that are already in a 3D environment.
There's already a wide variety of Opentrack plugins that use everything from off the shelf webcams to DIY infrared trackers to an iPhone app and FaceID/AirPods.
I can confirm that is works decently well with a sunny roof window in the background, which is normally enough for people to complain that my face is too dark.
8yo me, who instinctively tried to look behind the display's field of view during intense gaming sessions, would appreciate this feature very much. My belief is that if it shifted the pov to a lesser degree than in the demo, people generally wouldn't notice, but still subconsciously register this as a more immersive experience.
I'm also glad that the web version doesn't try to cook my laptop - good work.
If you click "Menu" and then "Settings" you can play around with e.g. the sensitivity. Ideally we'd automatically optimize the calibration according to for example what device you are using, but that's something we would do a bit more long-term.
I can see this quite useful for educational demonstrations of physics situations and mechanical systems (complex gearing, etc.). Also maybe for product simulations/demonstrations in the design phase — take output from CAD files and make a nice little 3D demo.
Maybe have an "inertia(?)" setting that makes it keep turning when you move far enough off center, as if you were continuing to walk around it.
The single-viewer limitation seems obvious and fundamental, and maybe a bit problematic for the above use cases, such as when showing something to a small group of people. One key may be to take steps to ensure it robustly locks onto and follows only one face/set of eyes. It would be jarring to have it locking onto different faces as conditions or positions subtly change.
Good ideas - we've considered these as well actually!
The intertia-idea wouldn't be too difficult to implement, but its usefulness would probably depend on the application area.
Yep exactly. Usually it locks onto one person's face but it can also jump around, so there are still optimizations we can do there - but generally it's supposed to be for one person. If you compare to VR headsets, two people can't wear the same VR headset anyway!
This is fun!
But I see it showing me a close-up view (smaller FOV) if I move my head back, and a wider view (larger FOV) if I bring my head closer. This is the opposite to what I'd expect: if I bring my head nearer to the screen, I should see a more detail, closer (narrower FOV).
Definitely! Our current focus is on Unity, because that's what we're most used to, but we'd also build the solution for at least Unreal and Godot as well!
The latency is unfortunately bad enough that it prevents me from getting any depth illusion :( I've seen other implementations where it works really well though.
May I ask what device you are trying it on? In our experience, it can vary depending on what device - something we're looking to improve on further down the line.
Do you remember which other implementations you've seen that worked really well?
Back in college (~2008) we implemented this with a 7 foot tall back-projected screen and a couple of Wii remotes after seeing Johnny Lee’s video. The nice thing with that screen was that you could stand so close to it you couldn’t really see the edges.
We had as many people come test as we could, and we found that 90% of them didn’t get a sense of depth, likely because it lacked stereo-vision cues. It only worked for folks with some form of monocular vision, incl myself, who were used to relying primarily on other cues like parallax.
really fancy idea and cool project! Its a bit stuttery probably due to the 30fps of a webcam but works. Feels slightly weird somehow though, probably from the lag.
update: just tried to open the site again now and its gone but leads to some kind of shop?
I tried implementing this with face detection-based head tracking after that demo (or maybe before; I can't remember). I got it working but the effect was very underwhelming. It looks great in that video, but it kind of sucks in real life.
I think the problem is in real life you have an enormous number of other visual cues that tell you that you're not really seeing something 3D - focus, stereoscopy (not for me though sadly), the fact that you know you're looking at the screen, inevitable lag from cameras, etc.
I can't view the videos because of their stupid cookie screen, but I wouldn't be too excited about this. The camera lag especially probably impossible to solve.
this (and several of his ideas) were the reason I value simple solutions so much in my work along with optimising for low cost. "If Johnny Lee can do this crazy thing in cheap, I can think something creative too"
Interestingly, for this parallax 3D effect to work, the head tracking needs to basically move "backwards" from typical head tracking since it needs to keep the focal point the same, if I'm understanding correctly. Any time I've tried this out it's fun and would likely be most useful for something like a 3D painting you hang on your wall.
Related idea, but not the same, might be my iOS and Android app that uses your phone's AR for head tracking and then sends that data to your PC for smooth sim game head tracking. https://smoothtrack.app
My company bought one of those Sony Spatial Reality Displays a couple of weeks ago and I have to say it is truly impressive, the 3D effect is really convincing.
I just tried this demo and it is cool, but nowhere as good.
From my experience this only ever looks good in a recorded video. Because we are used to assume that the real object captured within the video is perceivable in a stereoscopic form. But it‘s not. The real usage of head and viewtracking is as a controller, not to improve immersion.
One of the big issues with that phone was that in order to do dynamic perspective, you're having to run a 3D render at 60fps constantly. That's a huge power hog, and prevents you from doing many of the power savings techniques you otherwise could on a normal phone -- shutting down the GPU, reduced refresh rate, heck, even RAM backed displays.
> This content is hosted by a third party provider that does not allow video views without acceptance of Targeting Cookies. Please set your cookie preferences for Targeting Cookies to yes if you wish to view videos from these providers.
Yeah what is up? When i accept cookies i find myself at the bottom of the page at a checkout button with a dollar price number? This confused me, after three more tries I gave up.
Honestly I have no idea what the problem seems to be - we're not collecting any cookies. Also, there's no video, so I'm not sure what you are referring to...
Seemed to have some different names depending on region (Looksley's Line Up, Tales in a Box: Hidden Shapes in Perspective). I recall it working very well at the time.
imho this should be useful for driving/flight sims, giving the player the ability to lean inside the vehicle, changing their viewpoint on the surroundings.
How is this different from the many available head tracking devices? I used to play Flight Simulator with head tracking[1]. It was great, although it did consume a bit of CPU cycles. IR trackers are more efficient
But does it actually make stuff look like you can reach out and touch it like the 3D movies at theme parks? I don't see how it would. This just seems like parallax.
You can do this with a Kinect for head tracking and people do that with homemade pinball machines using a TV as the table, and a Kinect to track your head, so it looks like the table is 3D down into the TV.
But I don't think it creates the full 3D effect with things looking like they are coming way out of the screen and like a tangible thing you can reach out and touch.
Calling this 3D is a stretch, it does not make things appear like they're coming out of the screen. That requires having a way to show something different to each eye, something not possible with a standard display.
I got excited about this a few years ago when I was into digital pinball. I built an open source project called Phonemote using an iPhone to track eyes and relay telemetry over WiFi/BT and a plugin for FuturePinball.
However the result wasn’t that useful because we humans have 2 eyes.
The 3d effect is very compelling if you close/cover one eye.
But that becomes annoying quickly.
The best results I had was sneering a little hair gel in the center of my left eye glassses. Then it felt like I was using 2 eyes but really my right eye was seeing the pinball table clearly and fooling my brain.
63 comments
[ 2.6 ms ] story [ 103 ms ] threadWe built this because we wanted 3D experiences without needing a VR headset. The approach: use your webcam to track where you're looking, and adjust the 3D perspective to match.
Demo: https://portality.io/dragoncourtyard/ (Allow camera, move your head left/right)
It creates motion parallax - the same depth cue your brain uses when you look around a real object. Feels like looking through a window instead of at a flat screen.
Known limitations: - Only works for one viewer at a time - Needs decent lighting - Currently WebGL only
We're still figuring out where this is genuinely useful vs just a novelty. Gaming seems promising, also exploring education and product visualization.
Happy to answer questions!
First thing, there is no loading indicator and it takes too long to start so I though it was broken a few times before I realized I had to wait longer.
Second thing, although it was clearly tracking my head and moving the camera it did not make me feel like I'm looking into a window into a 3d scene behind the monitor.
These kinds of demos have been around before. I don't know why some work better than others.
some others:
https://discourse.threejs.org/t/parallax-effect-using-face-t... https://www.anxious-bored.com/blog/2018/2/25/theparallaxview...
And we will check out these links, appreciate you sharing it!
I'd say I'm not the only one who misses this technology in games, because a used New 3DS XL costs at least $200 on eBay right now, which is more than what I paid new.
I always thought 3D would combine really nicely with a ray traced graphics full of bright colors and reflections, similar to all those ray tracing demos with dozens of glossy marbles.
Fully agreed that if you want 100% full 6DOF immersion - go and pay hundreds or even thousands of dollars to wear a heavy and cumbersome headset on your head. We're not disputing that or thinking of competing with that.
What we're saying is that there may be a much larger market consisting of people who are not ready to commit to pay so much money to wear something that will give them motion sickness after 10 minutes.
If you're developing a VR game your market consists of 50 million people around the world who owns a VR headset. That's great. But since you already built the VR world in 3D, you could also open up the market to billions of people who want to play your game but on their own devices.
Admittedly, it won't be the same experience, but it could be a "midpoint". Not everyone can afford and is willing to pay for a VR headset.
Presumably developers could have combined this with parallax head tracking for an even stronger effect when you move your head (or the console), but as far as I know no one did.
What we're thinking is to enable this technology - as long as there is a camera and a screen - instantly accessible across billions of devices.
That means that the 3D effect would be applicable not only for games built for that specific console - but for any and all games that are already in a 3D environment.
Systems like trackir, which require dedicated hardware.
8yo me, who instinctively tried to look behind the display's field of view during intense gaming sessions, would appreciate this feature very much. My belief is that if it shifted the pov to a lesser degree than in the demo, people generally wouldn't notice, but still subconsciously register this as a more immersive experience.
I'm also glad that the web version doesn't try to cook my laptop - good work.
If you click "Menu" and then "Settings" you can play around with e.g. the sensitivity. Ideally we'd automatically optimize the calibration according to for example what device you are using, but that's something we would do a bit more long-term.
Appreciate it!
I can see this quite useful for educational demonstrations of physics situations and mechanical systems (complex gearing, etc.). Also maybe for product simulations/demonstrations in the design phase — take output from CAD files and make a nice little 3D demo.
Maybe have an "inertia(?)" setting that makes it keep turning when you move far enough off center, as if you were continuing to walk around it.
The single-viewer limitation seems obvious and fundamental, and maybe a bit problematic for the above use cases, such as when showing something to a small group of people. One key may be to take steps to ensure it robustly locks onto and follows only one face/set of eyes. It would be jarring to have it locking onto different faces as conditions or positions subtly change.
The intertia-idea wouldn't be too difficult to implement, but its usefulness would probably depend on the application area.
Yep exactly. Usually it locks onto one person's face but it can also jump around, so there are still optimizations we can do there - but generally it's supposed to be for one person. If you compare to VR headsets, two people can't wear the same VR headset anyway!
Do you remember which other implementations you've seen that worked really well?
We had as many people come test as we could, and we found that 90% of them didn’t get a sense of depth, likely because it lacked stereo-vision cues. It only worked for folks with some form of monocular vision, incl myself, who were used to relying primarily on other cues like parallax.
update: just tried to open the site again now and its gone but leads to some kind of shop?
update2: oh use the link in the comment for the demo: https://portality.io/dragoncourtyard/
I think the problem is in real life you have an enormous number of other visual cues that tell you that you're not really seeing something 3D - focus, stereoscopy (not for me though sadly), the fact that you know you're looking at the screen, inevitable lag from cameras, etc.
I can't view the videos because of their stupid cookie screen, but I wouldn't be too excited about this. The camera lag especially probably impossible to solve.
Related idea, but not the same, might be my iOS and Android app that uses your phone's AR for head tracking and then sends that data to your PC for smooth sim game head tracking. https://smoothtrack.app
I just tried this demo and it is cool, but nowhere as good.
https://www.youtube.com/watch?v=6trOg2IK2Zg
One of the big issues with that phone was that in order to do dynamic perspective, you're having to run a 3D render at 60fps constantly. That's a huge power hog, and prevents you from doing many of the power savings techniques you otherwise could on a normal phone -- shutting down the GPU, reduced refresh rate, heck, even RAM backed displays.
OK, that's a no then.
Would you be able to clarify?
https://youtu.be/4zZfsyHEcZA?si=BE2I991zEVxPEt9F&t=57
Seemed to have some different names depending on region (Looksley's Line Up, Tales in a Box: Hidden Shapes in Perspective). I recall it working very well at the time.
source code at https://github.com/guyromm/Window
imho this should be useful for driving/flight sims, giving the player the ability to lean inside the vehicle, changing their viewpoint on the surroundings.
[1] - https://www.youtube.com/watch?v=P07nIcczles (actually, this one was using a paper tracker because the face tracker had a big impact on fps)
You can do this with a Kinect for head tracking and people do that with homemade pinball machines using a TV as the table, and a Kinect to track your head, so it looks like the table is 3D down into the TV.
But I don't think it creates the full 3D effect with things looking like they are coming way out of the screen and like a tangible thing you can reach out and touch.
I did see a demo of 3D without glasses on a full monitor that DID make it look like it was coming out of the screen at CES, it requires a $3000 monitor though: https://www.3dgamemarket.net/content/32-4k-glasses-free-3d-g...
Also the 3DS obviously.
However the result wasn’t that useful because we humans have 2 eyes.
The 3d effect is very compelling if you close/cover one eye.
But that becomes annoying quickly.
The best results I had was sneering a little hair gel in the center of my left eye glassses. Then it felt like I was using 2 eyes but really my right eye was seeing the pinball table clearly and fooling my brain.