Amazing work! It was be interesting to create a 'spatial viewer' that could navigate around the object, rather than just watching the object rotate in front of you. If theres a background scene, then the object of interest could be extracted and three-dimensionalized, while keeping the background scene flat. Then the viewer could move around the object while maintaining (at least a flat version of) the background.
It looks like if you use the full pipeline, it takes 2.5 hours for textual inversion, 40 minutes for coarse estimation, and 20 minutes for fine estimation. For one image, on a 32G V100.
Very impressive work, but this reads more as data compression and retrieval algorithm than a generative model. For all of the examples they're inputting a novel 2D view of a known object and reconstructing the 3D mesh from the 2D/3D training set of the same object.
I'd like to see how it performs on a novel object that is similar to one in the training set, but not included in the training data.
It sounds to me like the novel view is generated from another algorithm - then their algorithm makes a mesh from these generated novel views to 'improve' the output in a two stage process (coarse -> fine).
Certainly a step above what a normal person could do in a 3D modeling program. The interesting thing about this type of stuff is that it would be harder for a professional to use those meshes as a base model; the topology doesn't lend itself to their process. For example -- the banana peels are puffy and two-dimensional. It would have to be completely restructured to be a convincingly peeled banana. So either the generated model has to be a finished product, or they are cost-effectively useless. CAD files are notoriously bad, and those are nice and mathematically easy to break down.
You're assuming that this is always part of a pipeline that ends with a clean mesh representation that can be rendered using traditional techniques.
I think it's equally likely that we'll end up with replacing meshes - or with a hybrid pipeline where non-mesh representations coexist with something else.
I am thinking about realtime mainly but I think the same thing might apply to "offline" rendering (does anyone still call it that?)
> So either the generated model has to be a finished product, or they are cost-effectively useless.
Not really. Depending on the use case and adequate tools it can be much faster to the alternative of making these manually as meshes and textures.
Using the banana as an example, if this can be converted to a volumetric model (voxels) the puffied side can be shaved off using sculpting tools and the model be converted to a mesh much faster than making it from scratch. While the end result wouldn't be good for looking at it up front, it can be perfectly viable for background props in a game, especially something that is viewed from a bird's eye view or a drawn out third person perspective (though even up front it'll look better than what you see in some games[0] - and that is AAA).
In fact there have been several games using photogrammetry already to construct 3D models out of taking photos of real places from various angles and converting them to point clouds and then to meshes - which after that they need to be cleaned up by artists. This all takes time, is costly and needs specialized hardware and software and yet developers do it. The linked paper is about a method that significantly lowers those barriers while giving decent results even if they still need to be edited.
It's pretty cool how quickly we're able to turn a 2D image into a 3D one these days. Although it's not perfect yet, you can see that it doesn't quite capture the depth of the horse and dragon sculpture near the edges. But this technology will definitely improve over time.
Now, here's a thought to chew on: once we've mastered 2D to 3D, what's stopping us from exploring 3D to 4D conversion? That would take this to an entirely new level.
This project seems to perform some rudimentary 3D to 4D conversion by rotating the object as a function of time. A first step, perhaps, on the path to inferring entire timelines.
Why are researchers focusing on single-image reconstruction? It seems like a party trick which isn't very useful, and pretty much impossible to reconstruct the original object accurately. It would be much more useful if many images from different angles could be used. Somewhat like NERF, but also predict missing views with 2D diffusion. Adding more images would get the model closer to ground truth.
Because there’s people working on all sorts of different problems and solving a problem in one area can apply better to some problems than others. Not to mention that solution approaches can often cross pollinate.
yeah, it's totally useless.... unless of course, you have only one image of the thing you're trying to model. and getting a second image will take millions of dollars or another trip around the earth from orbit.
They likely focus on the 1 image case because there it's the easiest to show visual progress over the competition. If you have 50+ images, the tech from 10 years ago is already good enough to get Hollywood-quality 3D scans.
From what I understand, they use the text keywords detected from the image as guidance and they also apply a loss between the current diffusion state and the source image. In effect, this is stable diffusion for 3D shapes but with clever conditioning. That means this algorithm will also work just fine if you have 2+ input images.
Do you have any source for scene reconstruction being hollywood-quality given 50+ images 10 years ago (assuming the images don't come from a controlled environment)?
I can find for example this from 2016: https://substance3d.adobe.com/magazine/go-scan-the-world-pho...
However, there's still a lot of manual work involved, and you don't easily get near-perfect PBR textures (which would be what I'd consider hollywood-quality).
I'd say the devil is in the details - if you want the best quality, you'd still need to control a lot of the environment.
The more assumptions you can bake into the parameters of some model, the more degrees of freedom you get in the actual measurement process (e.g. reducing the amount of actual data necessary).
True but 50+ images only work if they're good quality.
What would really make a difference if you could make a decent model from 30 images snapped by a drunk teenager on their $200 budget phone in bad lighting.
If money and resources are not an object then yeah it's easy but for most people it is.
It's not impossible, as you can find a good artist that can do it. If a human can do it, then there's at least a chance that ML can do it as well. Unless there is something about the problem space that I'm misunderstansing.
Party trick? To me this looks like a game changer for developing 3D art for videogames.
It's not there yet for AAA games, but will get there in time. In the meantime, right now as it is, it's could save days of time for the indie game dev.
Sometimes you want to construct a 3D model from a single image. E.g. Stable Diffusion is pretty good at generating images of original miniatures for gaming. Will be interesting to see how SD to Magic123 to 3D print will work for this.
Scale. You have to be able to do it once to be able to do it at all.
This is just for creative/artistic use, because you obviously don't want software to "guess" on something used for engineering. In that case, you would use an appropriate 3D scan.
Also, 3D assets aren't generally created as scenes. A 3D-2D-3D pipeline also has wide application in AR, VR, standalone modeling, animation, etc.
As long as the voxels are weighted by some sort of confidence value, generating a high quality 3D model based upon a dozen or so goodish quality models is a trivial fitting task.
The problem with the multi-perspective approaches are: to do it well you need lasers and extremely stable AND perfectly localized observation points, making it slow, expensive and fragile for robotic applications, OR the image-to-model component of it is a grotesque anti-physics artifact hallucination which reinforces entropy as much as it does valid data leading to shitty models.
Essentially, this single image to 3D step is the key to both forms.
Our minds build internal models from what we are seeing. What you perceive of the world is pulled from this model, which is "running" in mind and is apparently 3D.
To effectively mirror this process in machines, we need models that can extrapolate a 3D environment and it's objects from sensors. If we can get an approximation of what something looks like from a different perspective, it's valuable for naviatigation and objective planning.
We could also theorize about the effects it will have on machine's awareness and attention.
The banana has 2 bananas, and the bird has 2 beaks! Why is that? Does it not know about real life objects and what the image might be representing? Would it be possible to combine this with some other kind of model so it would be able to handle things like that?
NeRFs are cool technology that has its place and use. It is going to help with scene reconstruction and so on. And that is the reason why these CV researchers are flocking into using it. Despite knowing the limitations, they are giving their best to improve this technology. What I believe is NeRF is insufficient to be meaningful on its own. If you look into the architecture from this problem (the paper in the post), it clearly shows that they have some refinement phase going later on. A single RGB 2D to 3D model is such an ill-posed, we have to consider a lot of priors before diving into it.
There needs to be more foundational work in this field that can outperform or even improve the NeRF-based techniques. And the current herd mentality of researchers should be changed into exploring the alternatives. There is a reason why expensive automobile companies still rely on the physical modelling of their design. It's hard to simulate the physical conditions only through CAD modelling. Sure NeRFs are cool and they can make impressive results. That doesn't necessarily mean it is the means to an end. Look where rasterization brought us! NeRF is like rasterization. It is going to be used. But highly quality graphics was possible through GI and ray tracing! NeRF needs something equivalent that is physically grounded.
42 comments
[ 4.7 ms ] story [ 84.7 ms ] threadIt looks like if you use the full pipeline, it takes 2.5 hours for textual inversion, 40 minutes for coarse estimation, and 20 minutes for fine estimation. For one image, on a 32G V100.
I'd like to see how it performs on a novel object that is similar to one in the training set, but not included in the training data.
Gotta say, tools like these or NeRF will only revolutionise 3D modelling if they ever get topology right. That's the hard part.
I think it's equally likely that we'll end up with replacing meshes - or with a hybrid pipeline where non-mesh representations coexist with something else.
I am thinking about realtime mainly but I think the same thing might apply to "offline" rendering (does anyone still call it that?)
Not really. Depending on the use case and adequate tools it can be much faster to the alternative of making these manually as meshes and textures.
Using the banana as an example, if this can be converted to a volumetric model (voxels) the puffied side can be shaved off using sculpting tools and the model be converted to a mesh much faster than making it from scratch. While the end result wouldn't be good for looking at it up front, it can be perfectly viable for background props in a game, especially something that is viewed from a bird's eye view or a drawn out third person perspective (though even up front it'll look better than what you see in some games[0] - and that is AAA).
In fact there have been several games using photogrammetry already to construct 3D models out of taking photos of real places from various angles and converting them to point clouds and then to meshes - which after that they need to be cleaned up by artists. This all takes time, is costly and needs specialized hardware and software and yet developers do it. The linked paper is about a method that significantly lowers those barriers while giving decent results even if they still need to be edited.
[0] https://i.imgur.com/0OeNkZu.jpeg
Now, here's a thought to chew on: once we've mastered 2D to 3D, what's stopping us from exploring 3D to 4D conversion? That would take this to an entirely new level.
Research is additive not a zero sum thing.
lmao are you for real?
https://colmap.github.io/ Or, more modern: https://alicevision.org/
From what I understand, they use the text keywords detected from the image as guidance and they also apply a loss between the current diffusion state and the source image. In effect, this is stable diffusion for 3D shapes but with clever conditioning. That means this algorithm will also work just fine if you have 2+ input images.
The more assumptions you can bake into the parameters of some model, the more degrees of freedom you get in the actual measurement process (e.g. reducing the amount of actual data necessary).
But yes, the images totally came from a controlled environment. They rented like 50 similar cameras and hardware-synchronized the shutters.
What would really make a difference if you could make a decent model from 30 images snapped by a drunk teenager on their $200 budget phone in bad lighting.
If money and resources are not an object then yeah it's easy but for most people it is.
The best camera is the one you have with you, and the same goes for this too.
It's not there yet for AAA games, but will get there in time. In the meantime, right now as it is, it's could save days of time for the indie game dev.
Seems valuable to be able to generate 3d models from single 2d illustrations, for games and other media.
And there are probably heaps of other applications as well.
This is just for creative/artistic use, because you obviously don't want software to "guess" on something used for engineering. In that case, you would use an appropriate 3D scan.
Also, 3D assets aren't generally created as scenes. A 3D-2D-3D pipeline also has wide application in AR, VR, standalone modeling, animation, etc.
The problem with the multi-perspective approaches are: to do it well you need lasers and extremely stable AND perfectly localized observation points, making it slow, expensive and fragile for robotic applications, OR the image-to-model component of it is a grotesque anti-physics artifact hallucination which reinforces entropy as much as it does valid data leading to shitty models.
Essentially, this single image to 3D step is the key to both forms.
To effectively mirror this process in machines, we need models that can extrapolate a 3D environment and it's objects from sensors. If we can get an approximation of what something looks like from a different perspective, it's valuable for naviatigation and objective planning.
We could also theorize about the effects it will have on machine's awareness and attention.
There needs to be more foundational work in this field that can outperform or even improve the NeRF-based techniques. And the current herd mentality of researchers should be changed into exploring the alternatives. There is a reason why expensive automobile companies still rely on the physical modelling of their design. It's hard to simulate the physical conditions only through CAD modelling. Sure NeRFs are cool and they can make impressive results. That doesn't necessarily mean it is the means to an end. Look where rasterization brought us! NeRF is like rasterization. It is going to be used. But highly quality graphics was possible through GI and ray tracing! NeRF needs something equivalent that is physically grounded.
Have you seen Plenoxels? https://alexyu.net/plenoxels/
Next step: a few frames of video to produce a fully textured and articulated 3d model?