should work, and there are tons of new differentiable mesh and volumetric representations to try!
Aha thank you, this is helpful. Agreed there is much research needed to get this working but hopefully not too far off: https://twitter.com/kkpatain/status/1575758085821706240
Great question! Our team has been working on text-to-3d for ~1.5 years starting with https://ajayj.com/dreamfields. We had hoped that we could swap the contrastive CLIP model in Dream Fields for the generative Imagen…
it's a good idea :)
With smooth enough geometry converting NeRFs to meshes with marching cubes works pretty well. Would you say the topology of meshes on our website are still too incoherent for rigging?
I think you're both right! It is incredible that the 2D model knows enough about the visual world to produce many objects from all angles, but the 3D model is essential for gluing these views together, and in some ways…
just seems to work when the 3D model is simple and smooth
Yes, this is often a problem. We use view-dependent prompts (e.g. "cat wearing sunglasses, back view") but the pretrained 2D model often does not do a good job of interpreting non-canonical views and will put sunglasses…
Co-author here - we were also surprised :) The breadth of knowledge of the visual world embedded in these 2D models and what they unlock is astounding.
should work, and there are tons of new differentiable mesh and volumetric representations to try!
Aha thank you, this is helpful. Agreed there is much research needed to get this working but hopefully not too far off: https://twitter.com/kkpatain/status/1575758085821706240
Great question! Our team has been working on text-to-3d for ~1.5 years starting with https://ajayj.com/dreamfields. We had hoped that we could swap the contrastive CLIP model in Dream Fields for the generative Imagen…
it's a good idea :)
With smooth enough geometry converting NeRFs to meshes with marching cubes works pretty well. Would you say the topology of meshes on our website are still too incoherent for rigging?
I think you're both right! It is incredible that the 2D model knows enough about the visual world to produce many objects from all angles, but the 3D model is essential for gluing these views together, and in some ways…
just seems to work when the 3D model is simple and smooth
Yes, this is often a problem. We use view-dependent prompts (e.g. "cat wearing sunglasses, back view") but the pretrained 2D model often does not do a good job of interpreting non-canonical views and will put sunglasses…
Co-author here - we were also surprised :) The breadth of knowledge of the visual world embedded in these 2D models and what they unlock is astounding.