It's... neat? But I'm struggling to think of what the applications of this would actually be. 2D artwork usually doesn't have a consistent 3D space, which they acknowledge, but they don't seem to have overcome that problem in any useful sense. The scenes are barely coherent once they move from one of the originally drawn camera positions.
Both Futurama and Family Guy sometimes use 3d rendering for vehicles for example, and render it in a cartoon looking style and composit it with flat 2d animations.
Maybe similar kind of things could be an application of this.
Another possible use-case might be a game development studio developing a license game based on a 2d cartoon, but making the game 3d. They could use this as a tool for visualization while planning and developing, to iterate quickly and to reference how the original 2d could translate into 3d.
Not really? In those examples the hand crafted 3d assets already exist, this thing could at best recreate the 3d geometries the show creators made themselves. That seems useful mainly for cloning someone else's work.
If you were making 2D drawings with the intent of turning them into a 3D model then you would draw them to be coherent in 3D in the first place. The whole novelty of the research in the OP is that they're trying to reconstruct drawings that were never intended to make realistic sense in 3D.
Even if AI has a place in the 2D to 3D part of a pipeline, surely you'd still want the 2D artwork be unambigously representative of what the 3D asset should look like, rather than providing self-contradictory input data and praying that the AI can magically make it make sense.
A show that's being doing purely 2d art can't just integrate the 3d art in their pipeline on a whim. If they can, they probably already have the skills to make the model outright.
That assumes that consistency between viewpoints is actually desirable - part of the charm of 2D animation is that things can be stylized or exaggerated or simplified in ways that don't come naturally in a 3D workflow, where the "default" is for things to fit together realistically and any deviation from that takes additional effort.
If you do want numerous 2D artworks which share a realistically defined 3D space then that can easily be done by making a very rough 3D scene and then painting over it, you don't need any AI for that.
I think this is just a device used to demonstrate and advance the technology. I doubt this has a real application in this context given how little work is needed to 3D model these kinds of environments anyways.
A refined version of this could be used to make stereoscopic versions of cartoons.
On the other hand you are probably better off only using the depth prediction and filling any voids in using image generation instead of this mapping process.
SpongeBob brazenly violates 3D space rules (I mean, they also have fire underwater...). The writers and artists both draw heavy inspiration from Looney Tunes, where such rules are broken because it's funny to break them.
The ability to reconstruct a coherent 3d view from a sparse set of photos seems much more useful than for a set of 2d drawings of an entirely imagined space, I don't think 2d artists are cheaper than the 3d artists.
I was surprised by how poorly it reproduces the look from the perspective of specific images. For example, see the magic schoolbus further down. It feels like their algorithm could probably be tuned more in the direction of "trust the images".
A huge part of art is distinguishing between what "feels" right and what would be the case in reality. Even in the spaces I usually work in-- 3D animaton and film-- things in the background or maybe out-of-focus in the foreground or whatever are often distorted and weirdly juxtaposed to make something that looks right even if it wouldn't map to a real-world configuration that makes sense. 2D art is even less tied to real-world representations than that. What we can see in applications like this is how incredible our brains are at conceptually constructing ideas based on relatively abstract representations, and how incredible artists are at operating in the less-defined realms of that space. Maybe a scene seems to have a coherent perspective to the viewer, but the couch and end table in the BG were drawn as they would look shot with a 120mm lens while the foreground is deliberately claustrophobic and drawn like it was shot with a 30mm lens? It could look fine to us because we don't need to reason about the realistic 3D space those characters exist in-- we just need to understand that they're in a space like that because we know what it's like to be in spaces, and how people interact with them-- good art gives us just enough to communicate the core ideas making them the focus of the message, and lets our brains subconsciously make the connections and add all of the context to make a complete 'experience.' Everything is a potential layer of communication to achieve deliberate artistic effect-- the type of couch and end table, the often skewed or exaggerated scale and relationships between objects, etc.-- and it often just doesn't have a coherent real-world representation. Beyond that, in any given shot, things are certainly moved around to aid in composition, emphasize certain interactions, etc. etc. etc. If you notice it, then it's a continuity problem. If you don't notice it, then job well done. In the overwhelming majority of cases, nobody notices it, and we just happen to have a world where everything from every angle has really compelling composition.
An algorithm that needs to look at the lines and try to figure out a real-world scenario that correlates to that representation might be trying to create something that could never exist in any coherent form.
And the events of the cartoons would play on in a timed manner, so if you're not at the main point at the right time you could miss it. That would be cool.
Not if it looks anything like this... Honestly I'd be surprised if AI could do it justice. In a shot showing one character talking, panning around to see the other characters that AI pasted into the scene wouldn't be enough. Those characters would also have to be animated and show appropriate attention/reactions to what was being said/going on.
A little bit off topic, but related: are there any tools to which you can feed a few photos of a room from various angles and it will generate a floorplan or 3d model like this?
Yes, in fact at least one of them got funding from YC: Matterport.
There are many others: Kuula, Cupix, iStaging, EyeSpy360... Real estate companies use them a lot, e.g. to create a virtual tour for prospective buyers.
Yes but they don't just use a random set of photos. They take multiple 360 photos for the specific purpose of feeding them into their virtual tour app. That's why you can pan around.
Creating 3D spaces from inconsistent source images! Super fun idea.
I tried a crude and terrible version of something like this a few years ago, but not just inconsistent spaces without a clear ground truth - purely abstract non-space images which aren't supposed to represent a 3D space at all. Transform an abstract art painting (Kandinsky or Pollock for example) into a explorable virtual reality space. Obviously there is no 'ground truth' for whatever 'walking around inside a Pollock painting' means - the goal was just to see what happens if you try to do it anyway. The workflow was:
1. Start From Single Abstract Art Source Image
2. SinGan to Create Alternative 'viewpoints' of the 'scene'
3. 3d-photo-inpainting (or Ken Burns, similar project) on original and SinGan'd images (monocular depth mapping, outputs a zoom/rotate/pan video)
4. Throw 3d-photo-inpainting frames into photogrammetry app (Nerf didn't exist yet) and dial up all the knobs to allow for the maximum amount of errors and inconsistency
5. Pray the photogrammetry process doesn't explode (9 times out of 10 it crashed after 24 hours, brutal)
What is a technique/library that can take an image of a 3d environment/drawing of a room and detect a rough mesh highlighting ground, walls, barriers ?
> What is a technique/library that can take an image of a 3d environment/drawing of a room and detect a rough mesh highlighting ground, walls, barriers ?
Well just in case it wasn't obvious, Toon3D, the project being discussed, is doing that. Part of the workflow is asking the user to indicate correspondences between geometry in different images, and each image is processed individually to create blocks of geometry you can toggle on or off visually.
I can't think of a great application either. Maybe if you want to map camera movements when converting an animated scene from 2d to 3d. It'd probably be easier just to start from scratch though. Simple polygons with a toon shader would work for simpsons and family guy im sure.
I don't think it will ever catch fire in animation studio workflows. I can't see it beating the current process of applying toon rendering to 3D geometry. Though it may help renderers add variation to the output in a way that's more authentic and less random.
I'm wondering if it's at all useful in understanding / improving AI's ability to infer semantic meaning from even real images in a variety of scenarios? Like the ability to re-interpret an interpreted construction (drawing) of a scene.
One area of application may be helping machines better understand hand drawn human input?
"the current process of applying toon rendering to 3D geometry"
Is this widespread? My sense is that most mainstream TV animation that isn't obviously CGI is still drawn in 2D, with 3D work if used at all being relegated to backgrounds and the like.
I'm sure some marketer has an email chain open with a developer asking if they can use it to help advertise bigger houses to TikTok users who film at home, or something like that. Or maybe advertise luxury products to people who are in large homes.
Perhaps there are different types of kindness, because his films are deeply, profoundly kind; and acutely aware of inner life.
Someone with an extreme sensitivity for kindness can easily be seen as a curmudgeon by others, ie, after long years of disillusionment with the human race, or after a traumatic experience, or simply because of how they look.
Some people might be good at being kind 'in the moment', while others need to reflect - and the second kind can be a 'bigger', more encompassing, more effective or beneficial kindness.
And many (all?) of the people who give the most of themselves without hope for any reward genuinely care nothing for external validation or recognition - meaning we don't often hear about them or recognize them.
One could garner a reputation as an absolute arse, while accomplishing fantastically beneficial changes in the world. And conversely, a man could get a reputation as a folksy down-to-earth guy who you'd love to have a beer with, even as he sets the planet on a course to perpetual war. Cough.
Not pointed at @helloplanets, just need to note that he responded that way because it was an abomination. And they're all stockholm syndrome about the situation, "we can use AI to make grotesque monsters that feel no pain. All we want to do is make a machine that replaces human drawing." With this weird implied feeling he was supposed to congratulate them.
Quoting Miyazaki, which was not especially harsh given they showed him a naked mutant zombie crawling across the ground using it's head and arm as legs while constantly trying to arch it's butt toward the camera.
> Every morning, not recent days, but I see my friend who has a disability.
> It's so hard for him just to do a high five (waves hand showing difficulty)
> His arm with stiff muscle reaching out to my hand (demonstrates body stiffness)
> Now thinking of him, I can't watch this stuff and find it interesting
> Whoever creates this stuff has no idea what pain is, or whatsoever. I am utterly disgusted.
> If you really want to make creepy stuff, you can go ahead and do it
> I would never wish to incorporate this technology into my work at all
> I strongly feel that this is an insult to life itself.
Many YouTube comments seem to have understood this clip as a dismissal of AI in general. And, regardless of whether thats accurate, I disagree with this standpoint. It's not easy to defend this particular example. But seeing how Rainworld uses synthetic animation to simulate an alien, yet somehow familiar ecosystem, makes me excited for whats next.
From a Review by Matthewmatosis [1]:
> Not long after setting out, I found myself staying in a quiet place, just moving Slugcat around various obstacles as smoothly as I could. [...] What was happening on screen looked like an animal testing its limits so as to build survival skills. It was then that I knew that this system was a resounding success.
It's interesting that they used the Planet Express building from Futurama as one of their examples of 3D-inconsistency, because I'm pretty sure the exteriors are in fact computer-generated from a 3D model. Watch the show and you can see the establishing shots usually involve a smooth complex camera move around the building.
Agreed, most or all shots of the Planet Express building and Planet Express ship are 3D renderings, even in the original first few seasons. Beyond that, even some shots of Bender in Space are 3D renderings, especially in cases where a complex and continuous shift in perspective is required.
Non-photo-realistic (NPR) 3D art goes back a surprisingly long way in animations. I rewatched the 1988 Disney cartoon "Oliver and Company" recently, and I was surprised to see that the cars and buildings were "cel-shaded" 3D models. I assumed that the movie had been remastered, but when I looked it up, I found out that it was the first Disney movie ever to make heavy use of CGI[0] and that what I was seeing was in the original. The page I found says:
"This was the first Disney movie to make heavy use of computer animation. CGI effects were used for making the skyscrapers, the cars, trains, Fagin's scooter-cart and the climactic Subway chase. It was also the first Disney film to have a department created specifically for computer animation."
I guess it depends on the definition of "heavy use." I know in Tron a few scenes were CG, and there were a few CG+live-action bits, but the majority was filmed on normal physical sets in high-contrast, then painstakingly hand-processed[1] to add the neon "glow".
From your link:
>The 1982 Disney movie is privy to a remarkable number of firsts: the first feature-length film to combine CGI and live-action; the first talking and moving CGI character; the first film to combine a CGI character and a live-action one; the first fully CGI backgrounds… The list goes on and on.
>Eleven minutes of the film used "computer-assisted imagery" such as the skyscrapers, the taxi cabs, trains, Fagin's scooter-cart, and the climactic subway chase
But Disney financed and distributed Tron. It wasn't made by a Disney Studio, and most of the animation was outsourced to a Taiwanese studio because Disney wouldn't lend any of their own talent. So I think it's fair to say that Oliver & Company is the first Disney-made film to use CGI.
The Great Mouse Detective (1986) was earlier and the ending sequence is CG (printed out and traced onto cels so traditional 2D characters could be drawn on top).
That's a good point. What's funny is that "The Great Mouse Detective" was actually the film I was thinking of this whole time - I believe the ending sequence took place in Big Ben, and it looks quite good by 2024 standards. But I forgot the name of the movie and assumed it was "Oliver & Company" because Oliver is a plausible name for an English mouse :)
Kind of, it hasn't replaced anyone though. 3DCG just became good-enough basis for artists to build on, what AI bros have been fantasizing and advocating for couple years by now, yet completely ignored and mocked over.
Which tells, AI hatred don't necessary come from what pro-AI thinks where it comes from, people potentially just find AI art rage inducing.
Like, not even specific technical aspect of AI is bad or could use improvements. It just sits at the wrong side of the uncanny valley, and arguments clump around that.
Indeed many animated shows that don't look 3d animated have a 3d model somewhere in their pipeline these days.
Even if there's not a digital 3d model, there might be a physical model of the main locations in the studio for animators to refer to.
Isn't a lot of 3D in shows and games "faked" to look good to the viewer?
I remember seeing this blog write up on what 3D animators do to make things look acceptable. Like make a character 9 feet tall because when the camera panned them, they looked too short at their "real" in-system height. Or archway doors that are huge but at the perspective shot, look "normal" to us. Or having a short character stand on an out-of-scene blue box to make them having a conversation with a tall character not look silly due to an extreme height difference? Or a hallway that in real life would be 1,000 feet long but looks about 100 in-world because of how the camera passes past it, and how each door on that 1,000 foot hallway is 18 feet high, etc.
I wonder if shows like Futurama used those tricks as well, so when you sort of re-create the 3D space the animators were working in by reverse engineering like this, then you see the giant doors and 9 foot people and non-Euclidian hallways, etc. Just because it looks smooth as the camera passes it, doesn't mean that actual 3D model makes sense at other perspectives.
I don't have a ton of experience in this realm but from what I've seen it does happen a lot -- looking good is often better than being right. A great example of this is the way they tilted the models for Zelda's A Link Between Worlds[0]. Basically everything in the world is tilted back so it looks better for the camera angle, which is designed to mimic the feel of A Link to the Past.
I saw some video on A Difficult Game About Climbing a while back. The things they did to make the guy appear to grip the rocks and suck normally make the hands utterly bizarre when seen from the side.
the exteriors aren't generated from a 3D model, they are generated from many 3d model(s), of the same thing, that perhaps changed over time or changed between scenes, like the models on the star trek enterprise
I'm not a historian, but I remember a tour guide in Forum Romanum mentioned that current state of knowledge about how buildings and parts of cities looked like stems from their depictions on coins that period. Perhaps it could be used for that?
That'd be a small enough sample size that it would make sense to just have a human agonize over it.
I feel like this type of thing best applies in the kind of domain they're already in— TV shows with hundreds of hours of content that a machine can comb through looking for reference images to synthesize into these models.
In the past after I got Quest 2 and started to dive into the world photogrammetry. I went into the entire pipeline into building a 3D *model* from photos of an object taken from different angle. Pipeline involved using MeshRoom and few other software to clean up mesh and port it into Unity.
In the end (from my superficial) understanding, the problem with porting anything into VR (say in Unity in which you can walk around an object) is the important of creating a clean mesh. The 3D model that tools such as OP (I haven't dived deep into it yet) is these are point cloud in 3D space. They do not generate a 3D mesh.
Going from memory from tools I came across during my research, there is tools like this https://developer.nvidia.com/blog/getting-started-with-nvidi..., again, this does not generate a mesh. I think it is just a video and not something you can simply walk around in VR.
My low key motivation was to make a clone/model like what Matterport and sell it to real estate companies. Major gap in my understanding - the cause of me to loose steam is - I was not sure how are they able to automate the step to generate clean mesh from bunch of photos from a camera. To me, this is the most labor intensive part. Later, I heard there are ML model that is able to do this very step, I have no idea on this tho.
Perhaps using Unreal + nanite + PCVR would be a better option? Nanite can handle highly complex meshes and algorithmically simplify them in realtime. Basically a highly advanced LOD system. Not sure what limitations are but it's worth a try. Also I highly recommend using Reality Capture for photogrammetry. The pricing is super cheap and you pay per scan.
NeRFs are sort of last year's technology. The latest hype is about gaussian splats.
My understanding is that essentially these technologies take some images as input, and then train a model, where the model is learning the best way to render the imagines into a model in some sense. I think for gaussian splats, it represents images as sort of "blobs" in space, and each image has the same set of blobs that have to be used from some perspective to render the image, hence by positioning the splats such that each image is rendered correctly, you can reproduce the scene.
This training is currently very expensive and has to be done for each model, but produces an output that can be explored in real time.
I think the photogrammetry approaches used by matterport et all are older and require much higher quality input data, whereas the newer approaches can work with much less and lower quality data.
https://github.com/3DTopia/OpenLRM (They mention NeRF as inspiration but it seems original paper it was based on decided to use visual transformers. the opensource version seems to use meta's dino as one of key components)
115 comments
[ 2.7 ms ] story [ 200 ms ] threadMaybe similar kind of things could be an application of this.
Another possible use-case might be a game development studio developing a license game based on a 2d cartoon, but making the game 3d. They could use this as a tool for visualization while planning and developing, to iterate quickly and to reference how the original 2d could translate into 3d.
Team of 2d artists draw the desired vehicles for the cartoon from two or three angles. Software like this makes a usable 3d model of it.
Even if AI has a place in the 2D to 3D part of a pipeline, surely you'd still want the 2D artwork be unambigously representative of what the 3D asset should look like, rather than providing self-contradictory input data and praying that the AI can magically make it make sense.
If you do want numerous 2D artworks which share a realistically defined 3D space then that can easily be done by making a very rough 3D scene and then painting over it, you don't need any AI for that.
While rough, these do look better than some implementations of the artwork for cartoon games.
On the other hand you are probably better off only using the depth prediction and filling any voids in using image generation instead of this mapping process.
I could see it being used to create a "scratch track" that human animators animate on top of. An aid to tweening.
An algorithm that needs to look at the lines and try to figure out a real-world scenario that correlates to that representation might be trying to create something that could never exist in any coherent form.
It doesn't seem like the OP comes even close to this though.
There are many others: Kuula, Cupix, iStaging, EyeSpy360... Real estate companies use them a lot, e.g. to create a virtual tour for prospective buyers.
I tried a crude and terrible version of something like this a few years ago, but not just inconsistent spaces without a clear ground truth - purely abstract non-space images which aren't supposed to represent a 3D space at all. Transform an abstract art painting (Kandinsky or Pollock for example) into a explorable virtual reality space. Obviously there is no 'ground truth' for whatever 'walking around inside a Pollock painting' means - the goal was just to see what happens if you try to do it anyway. The workflow was:
1. Start From Single Abstract Art Source Image
2. SinGan to Create Alternative 'viewpoints' of the 'scene'
3. 3d-photo-inpainting (or Ken Burns, similar project) on original and SinGan'd images (monocular depth mapping, outputs a zoom/rotate/pan video)
4. Throw 3d-photo-inpainting frames into photogrammetry app (Nerf didn't exist yet) and dial up all the knobs to allow for the maximum amount of errors and inconsistency
5. Pray the photogrammetry process doesn't explode (9 times out of 10 it crashed after 24 hours, brutal)
I must have posted an example on Twitter but I can't find the right search term to find it. But for example, even 2019 tier depth mapping produced pretty fun videos from abstract art: https://x.com/jonathanfly/status/1174033265524690949 The closest thing I can find is photogrammetry of an NVIDIA GauGAN video (not consistent frame to frame) https://x.com/jonathanfly/status/1258127899401609217
I'm curious if this project can do a better job at the same idea. Maybe I can try this weekend.
Well just in case it wasn't obvious, Toon3D, the project being discussed, is doing that. Part of the workflow is asking the user to indicate correspondences between geometry in different images, and each image is processed individually to create blocks of geometry you can toggle on or off visually.
Older projects:
https://github.com/sniklaus/3d-ken-burns
https://github.com/vt-vl-lab/3d-photo-inpainting
I believe there are some NeRF variants that do something like this from a single image as well, but I haven't personally tried any.
I'm wondering if it's at all useful in understanding / improving AI's ability to infer semantic meaning from even real images in a variety of scenarios? Like the ability to re-interpret an interpreted construction (drawing) of a scene.
One area of application may be helping machines better understand hand drawn human input?
Is this widespread? My sense is that most mainstream TV animation that isn't obviously CGI is still drawn in 2D, with 3D work if used at all being relegated to backgrounds and the like.
So, not hyperbole.
Someone with an extreme sensitivity for kindness can easily be seen as a curmudgeon by others, ie, after long years of disillusionment with the human race, or after a traumatic experience, or simply because of how they look.
Some people might be good at being kind 'in the moment', while others need to reflect - and the second kind can be a 'bigger', more encompassing, more effective or beneficial kindness.
And many (all?) of the people who give the most of themselves without hope for any reward genuinely care nothing for external validation or recognition - meaning we don't often hear about them or recognize them.
One could garner a reputation as an absolute arse, while accomplishing fantastically beneficial changes in the world. And conversely, a man could get a reputation as a folksy down-to-earth guy who you'd love to have a beer with, even as he sets the planet on a course to perpetual war. Cough.
Quoting Miyazaki, which was not especially harsh given they showed him a naked mutant zombie crawling across the ground using it's head and arm as legs while constantly trying to arch it's butt toward the camera.
> Every morning, not recent days, but I see my friend who has a disability.
> It's so hard for him just to do a high five (waves hand showing difficulty)
> His arm with stiff muscle reaching out to my hand (demonstrates body stiffness)
> Now thinking of him, I can't watch this stuff and find it interesting
> Whoever creates this stuff has no idea what pain is, or whatsoever. I am utterly disgusted.
> If you really want to make creepy stuff, you can go ahead and do it
> I would never wish to incorporate this technology into my work at all
> I strongly feel that this is an insult to life itself.
(room sits in silence awkwardly)
From a Review by Matthewmatosis [1]:
> Not long after setting out, I found myself staying in a quiet place, just moving Slugcat around various obstacles as smoothly as I could. [...] What was happening on screen looked like an animal testing its limits so as to build survival skills. It was then that I knew that this system was a resounding success.
[1]: https://www.youtube.com/watch?v=x-Un2L5tF1w
Non-photo-realistic (NPR) 3D art goes back a surprisingly long way in animations. I rewatched the 1988 Disney cartoon "Oliver and Company" recently, and I was surprised to see that the cars and buildings were "cel-shaded" 3D models. I assumed that the movie had been remastered, but when I looked it up, I found out that it was the first Disney movie ever to make heavy use of CGI[0] and that what I was seeing was in the original. The page I found says:
"This was the first Disney movie to make heavy use of computer animation. CGI effects were used for making the skyscrapers, the cars, trains, Fagin's scooter-cart and the climactic Subway chase. It was also the first Disney film to have a department created specifically for computer animation."
References ----------
0: https://disney.fandom.com/wiki/Oliver_%26_Company
https://m.youtube.com/watch?v=mix9rStOqoI
Now I am curious to watch it
Tron came out 1982, six years before Oliver & Company.
https://en.wikipedia.org/wiki/Tron
[1] https://filmschoolrejects.com/tron-costumes-glowing-effect/ Thanks legions of Taiwanese animators (:
Sounds pretty heavy to me.
>Eleven minutes of the film used "computer-assisted imagery" such as the skyscrapers, the taxi cabs, trains, Fagin's scooter-cart, and the climactic subway chase
I think Tron wins in terms of CGI
https://youtu.be/P5hHV2torG0?t=126
https://youtu.be/sGxLWXtt6EQ?t=73
https://en.wikipedia.org/wiki/The_Rescuers_Down_Under
Wait, you're telling me that computers have enabled us to have fewer artists and thereby replacing artists for a long time now?!
Just like pretty much every industry out there?!
And that it's widely accepted so long as people get their cheap plastic goods from China?!
And that the current outrage won't even be remembered in 20 years?!
Which tells, AI hatred don't necessary come from what pro-AI thinks where it comes from, people potentially just find AI art rage inducing.
Like, not even specific technical aspect of AI is bad or could use improvements. It just sits at the wrong side of the uncanny valley, and arguments clump around that.
That's the real problem with generative AI.
I remember seeing this blog write up on what 3D animators do to make things look acceptable. Like make a character 9 feet tall because when the camera panned them, they looked too short at their "real" in-system height. Or archway doors that are huge but at the perspective shot, look "normal" to us. Or having a short character stand on an out-of-scene blue box to make them having a conversation with a tall character not look silly due to an extreme height difference? Or a hallway that in real life would be 1,000 feet long but looks about 100 in-world because of how the camera passes past it, and how each door on that 1,000 foot hallway is 18 feet high, etc.
I wonder if shows like Futurama used those tricks as well, so when you sort of re-create the 3D space the animators were working in by reverse engineering like this, then you see the giant doors and 9 foot people and non-Euclidian hallways, etc. Just because it looks smooth as the camera passes it, doesn't mean that actual 3D model makes sense at other perspectives.
[0]: https://www.gameinformer.com/b/news/archive/2013/11/20/the-t...
I feel like this type of thing best applies in the kind of domain they're already in— TV shows with hundreds of hours of content that a machine can comb through looking for reference images to synthesize into these models.
In the end (from my superficial) understanding, the problem with porting anything into VR (say in Unity in which you can walk around an object) is the important of creating a clean mesh. The 3D model that tools such as OP (I haven't dived deep into it yet) is these are point cloud in 3D space. They do not generate a 3D mesh.
Going from memory from tools I came across during my research, there is tools like this https://developer.nvidia.com/blog/getting-started-with-nvidi..., again, this does not generate a mesh. I think it is just a video and not something you can simply walk around in VR.
My low key motivation was to make a clone/model like what Matterport and sell it to real estate companies. Major gap in my understanding - the cause of me to loose steam is - I was not sure how are they able to automate the step to generate clean mesh from bunch of photos from a camera. To me, this is the most labor intensive part. Later, I heard there are ML model that is able to do this very step, I have no idea on this tho.
Are you saying it can take a point cloud 3D representation into a fully working and clean 3D mesh for VR?
My understanding is that essentially these technologies take some images as input, and then train a model, where the model is learning the best way to render the imagines into a model in some sense. I think for gaussian splats, it represents images as sort of "blobs" in space, and each image has the same set of blobs that have to be used from some perspective to render the image, hence by positioning the splats such that each image is rendered correctly, you can reproduce the scene.
This training is currently very expensive and has to be done for each model, but produces an output that can be explored in real time.
I think the photogrammetry approaches used by matterport et all are older and require much higher quality input data, whereas the newer approaches can work with much less and lower quality data.
https://github.com/3DTopia/OpenLRM (They mention NeRF as inspiration but it seems original paper it was based on decided to use visual transformers. the opensource version seems to use meta's dino as one of key components)