Early reviews indicate it is, as some reviewers have had access to spatial video taken from a phone, but I’m not sure if those were ideal conditions or just ad-hoc.
Pure speculation: when combined with the LIDAR depth sensor, the two cameras probably don't need as much physical separation to accurately create a depth map. The bigger problem is the inpainting needed to generate hidden detail when the movie is viewed from angles that are different from the one it was actually filmed from.
I was wondering about the use of the lidar sensor. Notably they do not say they are using it, but maybe they just wanted to keep it simple? Idk seems weird not to use lidar but also seems weird not to mention it if they are using it.
But if you have eyes 50mm apart, and source material from cameras 15mm apart (plus other depth information), you'll need to in-paint a small amount where your eyes could see "around" something and the cameras can't.
Or you can make do with 15mm-apart worth of "around"?
It's still more "spatial"/3D than a regular (single lens) image.
Plus this has a wide lens and a "regular" lens (actually both wide iirc but one is ultrawide), so it's not like 2 equal lenses 50mm apart like in regular stereoscopic "3d" video.
> Or you can make do with 15mm-apart worth of "around"?
You need to move the close objects further apart in left/right than they are in the camera. Then you need to fill the newly empty areas with something.
> Plus this has a wide lens and a "regular" lens (actually both wide iirc but one is ultrawide), so it's not like 2 equal lenses 50mm apart like in regular stereoscopic "3d" video.
> The bigger problem is the inpainting needed to generate hidden detail when the movie is viewed from angles that are different from the one it was actually filmed from.
My understanding is that very few consumer lidar sensors work well in daylight. It's hard to send out & detect significantly meaningful pulses of light, when there's sunlight all around.
I have an Intel L515 which is pretty remarkable in that sometimes you can get some depth finding outdoors. This is just a hobby item for me, I'm not an expert, but this launched as a fairly impressively long range & capable $350 USB3 system, and it seems like the market hasn't much comparable to it. Phones certainly I'd expect to be significantly worse.
>My understanding is that very few consumer lidar sensors work well in daylight. It's hard to send out & detect significantly meaningful pulses of light, when there's sunlight all around.
Aren't many "self driving car" sensors lidar? This would imply they can work in daylight - perhaps they don't necessarily depend on light on the sunlight spectrum?
(Or perhaps you don't consuder them consumer? Though those cars are consumer products, they're not made for military or industrial use)
Many cars are lidar,, but they use much stronger, bigger, and higher power lasers, on very expensive and precise rotator assemblies.
The L515 I mentioned was somewhat advanced at least for it's day because it used MEMS to steer its light source. That gave it leading class performance/size but it's still big and kinda hot-ish. Maybe we can keep scaling that kind of system performance to smaller sizes but even this package was pretty cutting edge & gave much better falloff than many competing systems, and was still largely an indoor sensor.
>The bigger problem is the inpainting needed to generate hidden detail when the movie is viewed from angles that are different from the one it was actually filmed from.
It's for spatial video, not for holographic video. When you see a 3d movie in a cinema, it's not like you can look at it from widely different anges and go peek from the side or behind the actors or whatever...
Given that iPhone cameras are ~2.5 mm apart, there needs to be some amount of in-painting when building the stereo image, so that it looks like it is taken with cameras that are ~6.5 mm apart.
One of the two cameras is the ultra-wide camera so it gets some additional parallax and visual information than just the separation to the other camera.
That’s not how parallax works. The wider field of view of the ultra wide camera will show some of the scene that the other camera doesn’t see, but over overlapping parts of the scene the parallax is a strict function of the location of the two lenses’ entrance windows.
They're probably fusing both lenses with the Lidar and some other tricks to reliably compute a dense surface. That would explain their suggestion not to move the camera very much, as that would cause a large portion of the mesh to be rebuilt. A blogger exported what appears to be two side by side videos, so maybe the view really is narrow or reconstruction happens at playback. There might also be Lidar data in there that he didn't notice.
Apple bought C3 Technologies a decade ago, and they use this technique to fuse photos from low flying charters to produce the 3d view in Apple Maps.
Two focal lengths at the same physical distance to the subject have exactly the same perspective (i.e. if you crop them to the same area they will look the same). There is no extra information to be had from that.
The depth information that can be obtained from differences in angular position/size of objects within cameras' FOV. There's a reason a photo taken with a 28mm doesn't look the same as with a 50 a few steps back.
Exactly. The steps back change perspective, not the lenses. That’s what I was trying to say above. In the iPhone both lenses are at the same distance to the subject.
Last part of the article. It states that they are viewed and shared as normal 2D videos on iphone, but when you connect a Vision Pro to your iCloud account, it can view your spatial videos in 3D.
Correct. Although it wouldn't surprise me if this _was_ locked into the apple ecosystem, it would be nice if it took advantage of existing open standards (whatever those may be).
I wonder if they'll have a way of converting to the old skool top/bottom for displaying on things like home TVs with 3D, or if they'll add update to AppleTV 4K for existing TVs.
Next is having them as sources in an NLE. Lots of ways to do that now, but that's been for separate L/R sources or top/bottom type of files. It's been so long since I've had to deal with that, I've not kept up with what the current state is for that world. Maybe they've all been updated to handle mv-hevc too??
There are folks who’ve written extractors for making SBS video from the mv-hevc streams but I haven’t seen a live playback conversion. I can’t imagine it’s difficult though if someone was motivated.
I also haven’t seen an example of an NLE supporting it, but I imagine we’ll see some within 2024? I’m more curious about how they might support playback in various headsets since a Quest 3 is now a lot cheaper than a 3D display or projector.
Thanks for the link. I tried it in my quest 3. The picture quality is awesome. I wonder why those "VR Experiences" are filmed for 180° which is super pixellated. High resolution + hdr + 3d is way more immersive then a super stretched out pixelated mess.
But the 3d effect is very subtle. I guess thats what you get from two cameras sitting 2cm next to each other (instead of the typical 6cm of two eyes). IMO the early reports from people trying the vision pro have entirely oversold it. Yes, it's better then your typical 180°, but the 3d is disappointing.
The samsung galaxy and pixel pro with their 3 lenses next to each other, using the two outer ones could do much better, if they implemented something similar. So in typical apple fashion, they are hamstrung by their own design.
> the early reports from people trying the vision pro have entirely oversold it
You can bet those demos were not recorded by an iPhone, but a pair of studio cameras spaced further apart. There were reports of apple working with movie studios to develop new hardware. We might still get that quality from professional productions.
"Spatial videos are captured in 1080p at 30 frames per second in standard dynamic range."
Won't this look extremely soft and stuttery on a Vision Pro? Even on a standard screen these videos will look quite bad compared to the 4k60 recordings that every Apple phone since the iPhone 8 can record.
Stutter and judder aren’t the same, but the whole point of that analysis is that 24p without judder does not feel jumpy, so you want a TV that can play it as intended.
Unless we're talking about different things - it already exists. There's two parts to it - estimating a depth map ("monocular depth estimation") and inpainting the occluded parts.
They have a lidar. So they have very accurate depth. This gives some limited possibility at extrapolating and showing things from different viewing angles. I think this is also how they can give a good stereo experience from cameras that are closer than the average human IPD.
And while stereo video as a format is simple, IMHO it’s still quite a feat of engineering to capture it from the current phone generation. There have been stereo Android phones before but they were engineered from the get go with stereo in mind, and sacrificed other elements to achieve it
I don't know how it works, I'm not opticist, but I assume there's some kind of math involved. Or maybe it diminishes the 3D effect but it's still enough to look interesting.
You can literally read the article- where Apple even put in a freakin picture showing which cameras are used.
And then you go on blithely accusing people telling you of refusing to participate (which is itself a logical fallacy) in a high handed, holier-than-thou manner.
Pro-model iPhones use lidar for depth video since the 13. It powers the portrait mode and background removal. Apps can do 3D scans of rooms or objects.
I know what LiDAR is (wrt ‘Apps can do 3D scans’) but I would like to see references for the parents claim. Or to its use in videos, portrait mode and background removal. There is a lot of misconceptions about it.
Just because it's the highest PPD doesn't mean the resolution is actually all that high.
Resolution has always been my main complaint about VR headsets and they are nowhere near the PPD needed to simulate watching high resolution content on a high resolution screen in front of you yet.
The Vision Pro is supposed to have 23MP, and an aspect ratio of 1.165:1 which suggests 3660 pixels across each eye.
However, what most people fail to think about is the FOV. The headset's entire FOV is about 100 degrees horizontally. THX standards recommend around 35 degrees for video horizontal FOV, but if people like a really immersive FOV for their video I think it's safe to push this up to around 50 degrees.
This means that a virtual theater screen within your FOV on the headset has only half the headset's resolution to work with. So that makes the width of a virtual 50 FOV horizontal screen 1830 pixels wide, which is less than even 1080p video at 1920 pixels wide.
And this is probably a bit high too as there is typically some amount of pixels you can't even see through the lens of a VR headset because you don't normally want to be able to see the edge of the panel inside the headset. Rather you see the edge of the optics. So that 3660 panel width is probably more like 3460, then divide that in half to 1730 pixels wide for your virtual screen. And that's if you like a 50 FOV wide virtual screen. That would be like sitting 5 feet away from a 65" TV which is arguably closer than the vast majority of people sit.
So if you are watching video at a pretty immersive 50 degree FOV, in an Apple Vision Pro, the native resolution of the video needs to only be about 1730x973 (assuming 16:9 video) at most to max out the PPD of the headset for reasonably sized virtual video screens.
That’s all well and good (and good info) but you’re not factoring in pixel alignment. There’s loss because you need to account for the video being at a non-aligned offset at any given view from the native screen pixels, and the video may be presented at different planar angles or distances from the user.
Therefore there would be some level of interpolation causing loss of clarity. In general, you’d want to double resolutions as a form of super sampling.
The 1080p here would be just about cutting it then at the initial 35 degrees you mentioned.
I think for the natural FoV of the phone camera and the types of content people tend to shoot (families etc) that a 50 degree view might be uncomfortably large. Perhaps for further afield content like landscapes etc
I guess I'm not sure why the video should need to be higher resolution for super-sampling than the pixel space it will be rendered on.
Why it would need to be higher resolution than the native pixel space that the virtual screen will be rendered over? Changing the angle or rotation of your head to the virtual screen only makes the virtual screen technically even lower resolution, and would make it so that you would notice an even lesser difference between higher and lower resolution videos being played on that virtual screen.
I chose 50 FOV because I think it's reasonable near upper-bound. I am heavily into home theater, and I have tried various VR headsets to replicated a home theater projector setup. Personally I am a fan of sitting to my projector (or TV) screens where the width of the screen is equal to my viewing distance. This results in a horizontal FOV of 53.1 degrees.
Now I often invite people over to my theater, or travel around and visit other people's HTs and I have found that most HT enthusiasts have no problems with this rough screen width-to-distance FOV for viewing movies.
However maybe HT enthusiasts are not very representative of the masses, but anyway I was only using it as more of an upper-bound and what I have been looking for in a headset.
If I want a virtual HT screen that has the native resolution of at least 1080p, and the headset has over 100 degree FOV (which most do), then I see that I need a headset to have more than 4000 pixels wide per eye, which sadly just doesn't exist yet.
But, at least in my experience, I don't think the resolution of the video itself needs to be substantially higher than the pixel space of the virtual screen. I think that regular real-time machine learning video up-scaling techniques are sufficient enough these days to render the video in a way that wont cause bad artifacts when the virtual screen pixels are skewed because the headset wearer is tilting or turning their head in relation to the position of the virtual screen.
Now I am fully aware that VR signal input also needs to be a higher resolution source than the native headset pixels, because there is some warping done to the image to make it look correct through the lenses, but that should only need to be applied to the 3D environment space the user is seeing. That means that the video being imposed into the virtual screen simply needs to be stretched via up-scale to a higher resolution and then this will be downscaled via the headset warp more or less back to its native resolution.
But what you are saying does make sense and I am sure is somewhat of a factor.
But then also, lower resolution video can still look quite fine once there is enough pixels to get good detail.
Your points are really good for home theatre playback,. However consider that in HT scenarios, you basically have your media and pixel array aligned in most cases. You essentially have a fairly straightforward interpolation before it goes to an analog space for viewing.
In VR you have two. You have the plane the video is played on and then the display. That disparity can cause issues like aliasing or generally weird bluriness.
You can see this even in relatively lower resolutin headsets like the Quest 2/3/Pro where even 1080p video often looks bad even though it’s occupying less than an actual 720p section of the display.
And yes you’re right you could employ an upscaler. But the name of the game in VR is latency and power draw. NVIDIA has done great work in this field but they have a lot more latitude due to form factor. To the best of my knowledge, Apple tend to prefer straight decode and playback without AI upscalers on their devices.
3D videos can be played on more devices than the vision pro and just because the vision pro has a high ppd that doesn't mean that the content will be rendering images with that high of a ppd.
Angle of the sensor also makes no difference other than aligned effective resolution. iPhone already crops in several modes also digitally stabilizes ie rotates. Worked with lots of dimensional photography measurements with Pixel. Nothing here is limiting really. Maybe there is some quality giveups. This isn’t film projector 3D where you lose the effect if your eyes aren’t aligned with the dual projectors. This is reconstructed 3D from two points of measurement. Do
You think the effect will only work if your phone is held 100% 90 degree level?
I mean your blowing me of smugly but all those points can be overcome with just a crop and tilt. It shows the typical Hackernews mentality “your wrong even though I have no experience in this space and only thought through this superficially.”
Again, I think you’re not thinking this through, and all your hubris of having experience in the space is a call to authority rather than any actual logical correctness.
1. Nobody would want to hold the phone at an angle to shoot all the time.
2. It’s already reducing resolution to 1080p because it’s cropping in on the Ultrawide when its axis aligned. How much lower would you go to do an angled shot? The resolution does matter because these will be playing back on one of the highest ppd headsets on the market.
Apple may want to sell more hardware, but that’s not particularly interesting insight. But at the same time, your theories of how it could be done on the 14 Pro would make for a rather poor user experience all around.
1. The censors do not need to be physically 90 degree aligned to get depth data. And the the crop can be rotated so you do not need to hold the phone at a dutch angle...
2. The censors are 24mp and 12mp, way way way above 1080p (2.1mp). A tilted crop sample will be no issue at all.
Also you are using the term 'appeal to authority' wrong as I am not referencing an outside authority here, I am referencing myself. I'm sorry I've made you a bit angry here, but just calm it down a bit and think it through. The overlayed images of two cameras are slightly mis-aligned, but the overlapping data is more than enough to produce a 2.1mp image. This can work both in traditional stereoscopic or with 3D generated depth data.
I think one thing that threw me though is the the advertising images for this are clearly deceptive. They show 3D generated video that can be viewed from multiple angles, when in actually HVEC is just stereoscopic.
Worked for 10 years on hardware/software integration design with Apple and Google. I am not ignoring the actual use case, I think you just aren't exactly able to conceptualize how the application of how this works. You use the camera normally, the software does all the crop/rotation/overlapping augmentation -- just like the smooth zoom between cameras already does. User knows nothing. Only issue would be if crop overlaps lens positions can't be normalized. But we already did that with smooth zoom.
Even if we throw out all the technical considerations, you’d still be asking the user to hold the phone diagonally to get the stereo alignment.
If instead, you said “hold it normally and we crop/rotate for you” , you would now have two point of views at different heights, you’d have very uncomfortable depth perception.
There’s not a solution here that would work with the 14 Pro camera layout unless you used the telephoto and then cropped the wide angle to match.
In which case you now have the issue that your users are stuck in effectively a 70mm fov which isn’t very useful for most use.
Again, I think you’re ignoring many aspects of the product design to try and say it’s possible.
Yeah I didn’t realize either till I saw a tear down. The only time I really notice in use is when I change the active lens and the perspective slightly shifts.
In a former life, I spent quite a bit of time shooting 3D with various rigs, but ultimately, the resulting video was just never anything much more than gimmicky. Without 6 DoF type of movement, the 3D just had the look of flat objects moving around on flat planes. It's just 2.5D animation in realtime. So much work and effort goes into getting it shot correctly, posting it correctly, delivering it correctly, and then viewing it correctly. All for such meh result.
It would work better if the cameras were further apart, about 6cm, giving the same parallax as the average human eyes. Right now the system has to make up what is behind objects in the picture as it can't really see for rea.
Yeah that would work well. In fact it's a little too far off. But AI could bring the view point back together and in that case the furthest possible distance would be best indeed (to get the most parallax)
Cool, this one is real 3D, looks like. Ever heard about any content made in this format? VR games are, well, games - we know they can do actual 3D just fine
Shameless plug - we recently released our $80 headset with support for spatial videos. It's a plastic headset, with two bluetooth controllers, that you put your phone into 'Google Cardboard style'. No need to wait for Vision Pro :-)
https://www.zappar.com/zapbox/
Thanks :-D Yes - our Gen 2 model ships internationally and our site hasn't quite been updated yet. If you drop us a message at support@zappar.com we can get one to you.
Good on you for the product.
Yes please update the site. I want to see if the ability to take spatial videos actually appears in local (NZ) iOS* before committing, and sending emails is a right pain versus a buy-now button
"Some features may not be available in all regions...." is appearing on my update text.
Yeah we've been thinking about a headset-only version too :-) The advantage of the controllers is they have 'Quest compatible' inputs - so content built for Quest can be ported with minimal fuss. That said there's many use cases that don't need the controllers, or for which hand tracking is sufficient, so we're definitely looking into it!
The app we provide allows you to view video (including spatial video) from your Photos library, but any app with a 'Google Cardboard' mode (showing left eye content on the left of the screen and right on the right) should work. We are looking at supporting other sources in our app but in some cases there are legal constraints, e.g. Google don't allow custom players to play YouTube videos.
Is the video passthrough offset to the right/left due to the camera location or are you able to correct that in app (asking due to the open peripheral vision).
Thank you! We do correct for this in the app - it's quite a complicated process because we only have a single 'monocular' camera image which we have to compose with 3D content. We use some tricks to render the camera in a way that feels comfortable with the content. We consider the camera passthrough to compete very favourably with Quest 3 / Pro and have also received independent feedback to say so, primarily because it's high resolution and high framerate (60 fps while we believe the Quest passthrough is only 30).
To clarify, are users able to perceive depth from the video passthrough? I'd imagine you'd need to adapt that monocular video into a stereoscopic 3d scene using lidar/etc. Or is it more of other tricks that make the 2D passthrough comfortable when viewed with the 3D AR content?
Hi - I'm Simon, and I've been leading the development on the low-level runtime bits for Zapbox.
We synthesise the views for the left and right eyes by projecting the camera image onto some assumed world geometry. Our 6-DoF experiences are generally designed for a table-top setting, and we project the camera onto an infinite plane on that table surface (there's a physical origin marker to define the origin of the space for the 6-DoF experiences). There's also a plane parallel to the device a few meters in the distance (to avoid a "horizon").
Sounds simple but for table-top experiences like viewing 3D models on a table it does give a convincing stereo effect - things like shadows from the model on the table just look right. Combined with the really low-latency display pipeline[1], the nice camera (1440x1080p @ 60fps), and the direct peripheral view of the real world thanks to the headset design gives a really pretty comfortable MR experience.
The next job will be improving the quality of the world geometry so that camera reprojection works better for static parts of the scene. We might try something live with lidar but for most rooms an offline process (or even manual placement and scaling of planes for walls / floors) would already provide a decent improvement.
The hardest part is foreground objects - hands, controllers, other people. Some scheme to detect and segment out those foreground bits and reproject them separately using an estimated depth map (whilst infilling the background layer for the static world) would be needed there. Complex in general but for things with known geometry ie controllers, it seems like something we should be able to do.
What's hopefully clear is that we're pushing to make the headset MR experience using your phone the best it can possibly be, given the limitations imposed by the hardware such as camera positioning. We're in a pretty good place already but there are definitely more improvements to come :)
[1] As this is HN, I can say a bit more technical stuff about the display pipeline... We have an iOS implementation for async timewarp - we've been able to circumvent the usual queuing of frames that happens on iOS to implement a late warp render just 3ms before a new frame starts scanning out. That means we can get a really recent camera frame (on iOS frames are delivered to the app around 20ms after capture) and also apply rotational reprojection for any movement since the frame was captured. Reprojecting for predicted translation at display time is on the roadmap too which should remove the last bit of noticeable latency when moving.
Thank you! The headset is supported on iPhone 11 and later - it's just the capture of spatial video that's only supported on the iPhone 15 Pro/Max. In case that helps with your purchasing decision ;-D
Insane. simple, elegant, succinctly explained, def. has a niche in the market, and affordable. kudos to you. Just annoyed I didn't think of it of and had the capacity to execute on something like this.
P.S.: If support for unity AR development is robust enough, I'd highly encourage you to aggressively pitch this to educational institutions. One major pain point is how challenging it is to provide a solution at scale to have students building in VR. A solution to be able to spend a modest amount of money, on something that is super robust, that leverages tech all the students have, and that I could get a dozen of for the price of two of the alternatives, would be a game changer.
Those are very kind words - thank you! It's been a labour of love for us over the last few years but we're at a point where this product really works now and so it's great to see the positive response. That's a really interesting point re education. I know the team here are very much looking at that vertical and will pass on your note. One of things that's nice about Zapbox is that the controller inputs are Quest-compatible, so educational content built for Quest should be easy to port, allowing for use cases where there are a small number of higher-cost headsets used in tandem with a larger number of Zapboxes.
For sure -- and I'd further emphasize that its a particular environment where the "downsides" of this approach would be strengths. For instance, a nearby school had a dozen quests for sporadic activities, but the fact that you can't easily peak at what the students were engaged in was a problem... well, with this, you can certainly do that! It's a space where I suspect the cost/ease/simplicity to fidelity tradeoff will be especially welcomed. Good luck with the product!
My guess from this magical talk is that this allows recording of stereoscopic video using two cameras with an additional LIDAR data stream that with processing could enhance the 3D depth effect when viewed in VR. 3D Video+?
Its nice to see all of this trickle out for a whole ecosystem
When I saw the Apple Vision headset, I thought it was a big miss to highlight it as a content creation device. The parent around their toddler with the unwieldy headset on needed to be ridiculued. I thought that should be a separate device, and just to let users discover on their own that the Apple Vision could technically record too.
It was great to see that the iPhone 15 Pro was going to be that device, and just a software update away! Here we are now
I called this when the vision Pro was announced [1]; so I'm going to take this opportunity to call something new (if you can tolerate my "bragginess" :)), even though both are not really crazy to predict. Apple is already producing new content for Apple TV+ in this spacial video format. It will probably be released along with the Vision Pro or before EoY 2024.
At the very least, Monarch Legacy of Monsters. Slow Horses -- which is an amazing series everyone should watch -- would benefit much less from this treatment.
There are some scenes in the latest episode of Foundation, particularly those in Trantor, where I found the shots/focus with "weird" focus and depths of field. I have a personal theory that those were already shot with spatial video capture, designed for Vision Pro in mind.
Tangentially, if you gave up on Foundation on S1 like a couple of my friends, S2 is much much better. And if you're a book purist they sort of explain why there are differences between the show and the books in a relatively decent way.
My prediction is that all of Apple's recent aerial videos that are available as screensavers were shot in 3D and will eventually be released as such. It's just too much of a coincidence that they released all that content when they did. It's absolutely perfect content for the headset.
It also explains the kinda-cool but kinda-weird content released by Disney such as the Star Wars Biomes and Vehicle fly-throughs. Expect those to be released in 3D alongside the headset.
Ultimately, we're going to be able to play HoloChess against R2-D2 inside the Millennium Falcon, stand on the bridge of a Star Destroyer, or examine Uncle Owen's charred remains on Tattoine.
Actually... you probably can scratch that last one. ;)
It took a few steps to get it out of the phone and onto YouTube in a way I could watch it in my VR headset, but the result is kind of nifty and nice to be able to produce with the phone I already have with me. I think a recent update to the Spatialify app that I was using makes it so you can do everything on the phone now (I had to run it through ffmpeg on my computer).
On iPhone and other devices, spatial videos appear as regular, 2D videos and can be watched or shared just like any other video
That's a shame. I would have expected the ability to have very subtle rotation/parallax effects, a la Facebook 3D photos (https://www.facebook.com/help/iphone-app/414295416095269). If you want people to actually start recording media to be viewed on the device en masse, I feel you need to throw them a bit of a carrot other than "well, you'll be able to look at them on some device you might own in years."
I feel that it would be very hard to add value to the video with something like this. For photos I think it makes more sense because they are static and the movement adds extra depth. However for video I think it would just seem shaky. I'm not even sure I love the picture version, although it is a nice option to have.
Casual book scanning could benefit from stereo cameras, if there's a way for iPhone apps to access this data.
As a mathematician I've long wanted to figure out the code to match up several photographs for a 3D model of an open book, to correctly flatten out the image of each page. Bonus points to use AI to clean up the image, or convert to source that faithfully represents the page layout.
There are apps that take one photo and then guess how to correct the distortion. They don't do as good a job as one could do with the actual 3D data.
Many of us don't want to dedicate the space or money for a professional book scanner, and can't find an affordable flatbed scanner that scans all the way to one edge to better support books. I'll sometimes slice a second used copy of a cookbook, to scan every page using a sheetfed scanner. Grabbing one's phone should win ease-of-use over any other solution, if it could match the quality of these other approaches. 3D cameras would help that.
Given that the distortion is always only going to be in the X dimension, unless the page is crumpled, could you just try to detect straight horizontal lines on the page, come up with the distortion to straighten them, and apply that to the whole page?
I've seen the poor performance of current phone apps. My point is that one could do better for less coding effort with access to a 3D model. Either a pair of photos, or a 3D camera.
I’d guess, without knowing the details, this is already how it is done? Guessing can only take you so far, not to mention some pages has no straight lines (and I think you need orthogonal lines for the reconstruction to be effective?) and some has deceivingly straight lines (or even slanted lines.)
What is described above is something I’ve seen in a documentary. They have a very fancy camera and the person holding the book can just slowly flipping through the pages, and it can reconstruct the flat pages.
I guess the problem is not technology or mathematical model behind it, but how to make it cheap. It is analogous to Google’s self driving car, that you want to solve the same problem with much cheaper technology.
I’m not sure if that data will be available on iPhone. Still, I understand that on the Vision Pro, none of this data (and similar data) will be available to developers due to privacy concerns.
That said, you might be able to do this already with iDevices equipped with LiDAR (e.g., iPhones Pro and iPads Pro). They’re already used to create 3D objects, but I suppose no one has used them yet to take the next step you are talking about.
I’ve started again. I stopped for a while because they stopped feeling novel, but when scrolling through older photos from 5+ years ago, the photos of people that had Live enabled bring me a lot more emotions.
I’ve noticed this too. The only reason I have live photos is because I went through a stretch of time where I accidentally took them, and now I wish more of my photos were.
I’ve enabled it by default, and I'm glad I did. It adds just a little bit more to the memory that was captured.
My only gripe is of the unrealistic “want my cake and eat it too variety,” where I shoot more in RAW nowadays and use third-party camera apps, so those don’t get the Live Photos treatment.
I’ve “solved” this by purposefully using the stock camera app for quick snaps of moments I want to be memorialized for sentimental reasons and reserving the RAW stuff for more artistic purposes, going even as far as taking another photo with the stock camera app if the situation calls for both.
But honestly, I much more appreciate my Live Photos, if only because they’re rolled into the Moments the iPhone automatically creates, which make for a great memory.
174 comments
[ 2.6 ms ] story [ 181 ms ] threadIt's still more "spatial"/3D than a regular (single lens) image.
Plus this has a wide lens and a "regular" lens (actually both wide iirc but one is ultrawide), so it's not like 2 equal lenses 50mm apart like in regular stereoscopic "3d" video.
You need to move the close objects further apart in left/right than they are in the camera. Then you need to fill the newly empty areas with something.
> Plus this has a wide lens and a "regular" lens (actually both wide iirc but one is ultrawide), so it's not like 2 equal lenses 50mm apart like in regular stereoscopic "3d" video.
This doesn't affect anything.
> The bigger problem is the inpainting needed to generate hidden detail when the movie is viewed from angles that are different from the one it was actually filmed from.
I have an Intel L515 which is pretty remarkable in that sometimes you can get some depth finding outdoors. This is just a hobby item for me, I'm not an expert, but this launched as a fairly impressively long range & capable $350 USB3 system, and it seems like the market hasn't much comparable to it. Phones certainly I'd expect to be significantly worse.
Aren't many "self driving car" sensors lidar? This would imply they can work in daylight - perhaps they don't necessarily depend on light on the sunlight spectrum?
(Or perhaps you don't consuder them consumer? Though those cars are consumer products, they're not made for military or industrial use)
The L515 I mentioned was somewhat advanced at least for it's day because it used MEMS to steer its light source. That gave it leading class performance/size but it's still big and kinda hot-ish. Maybe we can keep scaling that kind of system performance to smaller sizes but even this package was pretty cutting edge & gave much better falloff than many competing systems, and was still largely an indoor sensor.
It's for spatial video, not for holographic video. When you see a 3d movie in a cinema, it's not like you can look at it from widely different anges and go peek from the side or behind the actors or whatever...
Apple bought C3 Technologies a decade ago, and they use this technique to fuse photos from low flying charters to produce the 3d view in Apple Maps.
[ Paper: https://ui.adsabs.harvard.edu/abs/2008SPIE.6946E..0DI/abstra... ]
[ Coverage: https://9to5mac.com/2011/10/29/apple-acquired-mind-blowing-3... ]
[ Similar: https://web.stanford.edu/class/ee367/Winter2021/projects/rep... ]
Exactly. The steps back change perspective, not the lenses. That’s what I was trying to say above. In the iPhone both lenses are at the same distance to the subject.
But I thought the parent question was about watching the spacial videos — as 3D — on non-Apple VR devices (e.g. Quest).
Any platform can implement playback of them.
Next is having them as sources in an NLE. Lots of ways to do that now, but that's been for separate L/R sources or top/bottom type of files. It's been so long since I've had to deal with that, I've not kept up with what the current state is for that world. Maybe they've all been updated to handle mv-hevc too??
I also haven’t seen an example of an NLE supporting it, but I imagine we’ll see some within 2024? I’m more curious about how they might support playback in various headsets since a Quest 3 is now a lot cheaper than a 3D display or projector.
I followed [1] on a Pico 4 headset (video is about Quest 3) and it was pretty fun. The aspect ratio was a bit wonky.
[1] https://www.youtube.com/watch?v=SH96y2QjCGw
But the 3d effect is very subtle. I guess thats what you get from two cameras sitting 2cm next to each other (instead of the typical 6cm of two eyes). IMO the early reports from people trying the vision pro have entirely oversold it. Yes, it's better then your typical 180°, but the 3d is disappointing.
The samsung galaxy and pixel pro with their 3 lenses next to each other, using the two outer ones could do much better, if they implemented something similar. So in typical apple fashion, they are hamstrung by their own design.
You can bet those demos were not recorded by an iPhone, but a pair of studio cameras spaced further apart. There were reports of apple working with movie studios to develop new hardware. We might still get that quality from professional productions.
Won't this look extremely soft and stuttery on a Vision Pro? Even on a standard screen these videos will look quite bad compared to the 4k60 recordings that every Apple phone since the iPhone 8 can record.
https://www.rtings.com/tv/tests/motion/24p
The latter is boring and easy to implement, the former much more compelling and technically difficult.
When it comes to photography "AI" is a spice best used in moderation, cf eg https://www.theverge.com/2023/12/2/23985299/iphone-bridal-ph...
And while stereo video as a format is simple, IMHO it’s still quite a feat of engineering to capture it from the current phone generation. There have been stereo Android phones before but they were engineered from the get go with stereo in mind, and sacrificed other elements to achieve it
So I guess they're doing some processing to generate 3D models, or something along those lines.
I don't know how it works, I'm not opticist, but I assume there's some kind of math involved. Or maybe it diminishes the 3D effect but it's still enough to look interesting.
Stereoscopic video that has depth information courtesy of the LiDAR sensor.
From people that have used the headset it feels like you're actually there.
The iphone cameras are close together, is there really a benefit in recording both at once?
So, are you confirming they record from two lenses or just here for the reductio ad absurdum?
And then you go on blithely accusing people telling you of refusing to participate (which is itself a logical fallacy) in a high handed, holier-than-thou manner.
It has a slow startup time, limited range, and low resolution.
Possibly it could be used for depth calibration but I doubted it would be used for depth information.
Can you link any such review? Gruber is Apple's biggest shill and even he said that it's definitely not that immersive.
That’s likely necessary to shoot this format, as it uses the pair.
On a 14 pro, you’d have to hold the phone diagonally and even then the sensors wouldn’t align with the axis causing cutoff.
1. Clearly the array of cameras on these phones is along two axes, so the pedantic definition of two points on a line is irrelevant
2. You’d be shooting everything at a Dutch angle to maintain stereo if they weren’t aligned horizontally or vertically.
3. The sensors are rectangles and need to be aligned along their longest axes to maximize the most field of view.
The iPhone 14 could offer a lower FoV. The video format encodes the FoV of the recording already.
Bear in mind this has to play back on one of the highest ppd displays on the market when the Vision Pro releases.
Resolution has always been my main complaint about VR headsets and they are nowhere near the PPD needed to simulate watching high resolution content on a high resolution screen in front of you yet.
The Vision Pro is supposed to have 23MP, and an aspect ratio of 1.165:1 which suggests 3660 pixels across each eye.
However, what most people fail to think about is the FOV. The headset's entire FOV is about 100 degrees horizontally. THX standards recommend around 35 degrees for video horizontal FOV, but if people like a really immersive FOV for their video I think it's safe to push this up to around 50 degrees.
This means that a virtual theater screen within your FOV on the headset has only half the headset's resolution to work with. So that makes the width of a virtual 50 FOV horizontal screen 1830 pixels wide, which is less than even 1080p video at 1920 pixels wide.
And this is probably a bit high too as there is typically some amount of pixels you can't even see through the lens of a VR headset because you don't normally want to be able to see the edge of the panel inside the headset. Rather you see the edge of the optics. So that 3660 panel width is probably more like 3460, then divide that in half to 1730 pixels wide for your virtual screen. And that's if you like a 50 FOV wide virtual screen. That would be like sitting 5 feet away from a 65" TV which is arguably closer than the vast majority of people sit.
So if you are watching video at a pretty immersive 50 degree FOV, in an Apple Vision Pro, the native resolution of the video needs to only be about 1730x973 (assuming 16:9 video) at most to max out the PPD of the headset for reasonably sized virtual video screens.
Therefore there would be some level of interpolation causing loss of clarity. In general, you’d want to double resolutions as a form of super sampling.
The 1080p here would be just about cutting it then at the initial 35 degrees you mentioned.
I think for the natural FoV of the phone camera and the types of content people tend to shoot (families etc) that a 50 degree view might be uncomfortably large. Perhaps for further afield content like landscapes etc
Why it would need to be higher resolution than the native pixel space that the virtual screen will be rendered over? Changing the angle or rotation of your head to the virtual screen only makes the virtual screen technically even lower resolution, and would make it so that you would notice an even lesser difference between higher and lower resolution videos being played on that virtual screen.
I chose 50 FOV because I think it's reasonable near upper-bound. I am heavily into home theater, and I have tried various VR headsets to replicated a home theater projector setup. Personally I am a fan of sitting to my projector (or TV) screens where the width of the screen is equal to my viewing distance. This results in a horizontal FOV of 53.1 degrees.
Now I often invite people over to my theater, or travel around and visit other people's HTs and I have found that most HT enthusiasts have no problems with this rough screen width-to-distance FOV for viewing movies.
However maybe HT enthusiasts are not very representative of the masses, but anyway I was only using it as more of an upper-bound and what I have been looking for in a headset.
If I want a virtual HT screen that has the native resolution of at least 1080p, and the headset has over 100 degree FOV (which most do), then I see that I need a headset to have more than 4000 pixels wide per eye, which sadly just doesn't exist yet.
But, at least in my experience, I don't think the resolution of the video itself needs to be substantially higher than the pixel space of the virtual screen. I think that regular real-time machine learning video up-scaling techniques are sufficient enough these days to render the video in a way that wont cause bad artifacts when the virtual screen pixels are skewed because the headset wearer is tilting or turning their head in relation to the position of the virtual screen.
Now I am fully aware that VR signal input also needs to be a higher resolution source than the native headset pixels, because there is some warping done to the image to make it look correct through the lenses, but that should only need to be applied to the 3D environment space the user is seeing. That means that the video being imposed into the virtual screen simply needs to be stretched via up-scale to a higher resolution and then this will be downscaled via the headset warp more or less back to its native resolution.
But what you are saying does make sense and I am sure is somewhat of a factor.
But then also, lower resolution video can still look quite fine once there is enough pixels to get good detail.
In VR you have two. You have the plane the video is played on and then the display. That disparity can cause issues like aliasing or generally weird bluriness.
You can see this even in relatively lower resolutin headsets like the Quest 2/3/Pro where even 1080p video often looks bad even though it’s occupying less than an actual 720p section of the display.
And yes you’re right you could employ an upscaler. But the name of the game in VR is latency and power draw. NVIDIA has done great work in this field but they have a lot more latitude due to form factor. To the best of my knowledge, Apple tend to prefer straight decode and playback without AI upscalers on their devices.
That it can play elsewhere is not a consideration.
I mean your blowing me of smugly but all those points can be overcome with just a crop and tilt. It shows the typical Hackernews mentality “your wrong even though I have no experience in this space and only thought through this superficially.”
Apple wants to sell more hardware.
1. Nobody would want to hold the phone at an angle to shoot all the time.
2. It’s already reducing resolution to 1080p because it’s cropping in on the Ultrawide when its axis aligned. How much lower would you go to do an angled shot? The resolution does matter because these will be playing back on one of the highest ppd headsets on the market.
Apple may want to sell more hardware, but that’s not particularly interesting insight. But at the same time, your theories of how it could be done on the 14 Pro would make for a rather poor user experience all around.
2. The censors are 24mp and 12mp, way way way above 1080p (2.1mp). A tilted crop sample will be no issue at all.
Also you are using the term 'appeal to authority' wrong as I am not referencing an outside authority here, I am referencing myself. I'm sorry I've made you a bit angry here, but just calm it down a bit and think it through. The overlayed images of two cameras are slightly mis-aligned, but the overlapping data is more than enough to produce a 2.1mp image. This can work both in traditional stereoscopic or with 3D generated depth data.
I think one thing that threw me though is the the advertising images for this are clearly deceptive. They show 3D generated video that can be viewed from multiple angles, when in actually HVEC is just stereoscopic.
Like, sure you can do any number of things if you don’t consider how it’s used at all or what the user experience is.
If instead, you said “hold it normally and we crop/rotate for you” , you would now have two point of views at different heights, you’d have very uncomfortable depth perception.
There’s not a solution here that would work with the 14 Pro camera layout unless you used the telephoto and then cropped the wide angle to match.
In which case you now have the issue that your users are stuck in effectively a 70mm fov which isn’t very useful for most use.
Again, I think you’re ignoring many aspects of the product design to try and say it’s possible.
To be honest I look at the 3 cameras on the back of my 14 Pro and I have no idea which is which.
"Some features may not be available in all regions...." is appearing on my update text.
Now seeing it here, it feels like a real product all of a sudden!
Is the video passthrough offset to the right/left due to the camera location or are you able to correct that in app (asking due to the open peripheral vision).
We synthesise the views for the left and right eyes by projecting the camera image onto some assumed world geometry. Our 6-DoF experiences are generally designed for a table-top setting, and we project the camera onto an infinite plane on that table surface (there's a physical origin marker to define the origin of the space for the 6-DoF experiences). There's also a plane parallel to the device a few meters in the distance (to avoid a "horizon").
Sounds simple but for table-top experiences like viewing 3D models on a table it does give a convincing stereo effect - things like shadows from the model on the table just look right. Combined with the really low-latency display pipeline[1], the nice camera (1440x1080p @ 60fps), and the direct peripheral view of the real world thanks to the headset design gives a really pretty comfortable MR experience.
The next job will be improving the quality of the world geometry so that camera reprojection works better for static parts of the scene. We might try something live with lidar but for most rooms an offline process (or even manual placement and scaling of planes for walls / floors) would already provide a decent improvement.
The hardest part is foreground objects - hands, controllers, other people. Some scheme to detect and segment out those foreground bits and reproject them separately using an estimated depth map (whilst infilling the background layer for the static world) would be needed there. Complex in general but for things with known geometry ie controllers, it seems like something we should be able to do.
What's hopefully clear is that we're pushing to make the headset MR experience using your phone the best it can possibly be, given the limitations imposed by the hardware such as camera positioning. We're in a pretty good place already but there are definitely more improvements to come :)
[1] As this is HN, I can say a bit more technical stuff about the display pipeline... We have an iOS implementation for async timewarp - we've been able to circumvent the usual queuing of frames that happens on iOS to implement a late warp render just 3ms before a new frame starts scanning out. That means we can get a really recent camera frame (on iOS frames are delivered to the app around 20ms after capture) and also apply rotational reprojection for any movement since the frame was captured. Reprojecting for predicted translation at display time is on the roadmap too which should remove the last bit of noticeable latency when moving.
P.S.: If support for unity AR development is robust enough, I'd highly encourage you to aggressively pitch this to educational institutions. One major pain point is how challenging it is to provide a solution at scale to have students building in VR. A solution to be able to spend a modest amount of money, on something that is super robust, that leverages tech all the students have, and that I could get a dozen of for the price of two of the alternatives, would be a game changer.
When I saw the Apple Vision headset, I thought it was a big miss to highlight it as a content creation device. The parent around their toddler with the unwieldy headset on needed to be ridiculued. I thought that should be a separate device, and just to let users discover on their own that the Apple Vision could technically record too.
It was great to see that the iPhone 15 Pro was going to be that device, and just a software update away! Here we are now
[1] https://news.ycombinator.com/item?id=36225900
Tangentially, if you gave up on Foundation on S1 like a couple of my friends, S2 is much much better. And if you're a book purist they sort of explain why there are differences between the show and the books in a relatively decent way.
It also explains the kinda-cool but kinda-weird content released by Disney such as the Star Wars Biomes and Vehicle fly-throughs. Expect those to be released in 3D alongside the headset.
Ultimately, we're going to be able to play HoloChess against R2-D2 inside the Millennium Falcon, stand on the bridge of a Star Destroyer, or examine Uncle Owen's charred remains on Tattoine.
Actually... you probably can scratch that last one. ;)
It took a few steps to get it out of the phone and onto YouTube in a way I could watch it in my VR headset, but the result is kind of nifty and nice to be able to produce with the phone I already have with me. I think a recent update to the Spatialify app that I was using makes it so you can do everything on the phone now (I had to run it through ffmpeg on my computer).
That's a shame. I would have expected the ability to have very subtle rotation/parallax effects, a la Facebook 3D photos (https://www.facebook.com/help/iphone-app/414295416095269). If you want people to actually start recording media to be viewed on the device en masse, I feel you need to throw them a bit of a carrot other than "well, you'll be able to look at them on some device you might own in years."
As a mathematician I've long wanted to figure out the code to match up several photographs for a 3D model of an open book, to correctly flatten out the image of each page. Bonus points to use AI to clean up the image, or convert to source that faithfully represents the page layout.
There are apps that take one photo and then guess how to correct the distortion. They don't do as good a job as one could do with the actual 3D data.
Many of us don't want to dedicate the space or money for a professional book scanner, and can't find an affordable flatbed scanner that scans all the way to one edge to better support books. I'll sometimes slice a second used copy of a cookbook, to scan every page using a sheetfed scanner. Grabbing one's phone should win ease-of-use over any other solution, if it could match the quality of these other approaches. 3D cameras would help that.
I've seen the poor performance of current phone apps. My point is that one could do better for less coding effort with access to a 3D model. Either a pair of photos, or a 3D camera.
Something that could map the 3D surface onto a sheet would be awesome.
What is described above is something I’ve seen in a documentary. They have a very fancy camera and the person holding the book can just slowly flipping through the pages, and it can reconstruct the flat pages.
I guess the problem is not technology or mathematical model behind it, but how to make it cheap. It is analogous to Google’s self driving car, that you want to solve the same problem with much cheaper technology.
That said, you might be able to do this already with iDevices equipped with LiDAR (e.g., iPhones Pro and iPads Pro). They’re already used to create 3D objects, but I suppose no one has used them yet to take the next step you are talking about.
This will give you more reason to buy a new iPhone with more storage or upgrade your iCloud account. Memories will be made and bills will be paid.
This spatial/stereo/3D video format will die the same slow infamous death I guess?
https://www.samsung.com/au/support/mobile-devices/motion-pho...
My only gripe is of the unrealistic “want my cake and eat it too variety,” where I shoot more in RAW nowadays and use third-party camera apps, so those don’t get the Live Photos treatment.
I’ve “solved” this by purposefully using the stock camera app for quick snaps of moments I want to be memorialized for sentimental reasons and reserving the RAW stuff for more artistic purposes, going even as far as taking another photo with the stock camera app if the situation calls for both.
But honestly, I much more appreciate my Live Photos, if only because they’re rolled into the Moments the iPhone automatically creates, which make for a great memory.