Why use SDFs instead of NeRFs? Those were designed to be differentiable. Then you could turn the NeRF to an SDF later.
Related:
https://nvlabs.github.io/instant-ngp/
Nerfs don’t contain a surface representation, but instead contain a occupancy field to represent the shape. This has implications for rendering speed (you generally need more samples along each ray to get an accurate representation). It also has implications for uses outside of rendering, e.g. in physics simulations where you need to know exactly where the surface is. Lastly, you can use SDFs to get arbitrarily detailed meshes by sampling the field at some resolution, who which is trickier with nerfs because the surface is more fuzzy.
In the linked video presentation [1] they automatically turn the NeRF into a mesh, so you could do the same and get faster rendering speed than SDFs. I wonder if simply enforcing less transparency towards the end of the training in a trained NeRF would help with converging to sharper object boundaries and would get the benefits of fast training without the downsides. The linked paper doesn't even discuss training times which suggests it's really bad.
NeRFs a are also direction-dependent so work with specular surfaces too.
Why use NeRFs? An important question is: what is the difference? The instant NGP learns an SDF, so it isn’t an either/or question, right? A couple of people have mentioned to me that the Berkeley paper doesn’t require a neural network, that you can use an optimizer on the density field instead. I don’t fully grasp the exact difference between those things in detail, having not implemented it myself, but using a direct optimizer without a neural network seems like an important conceptual distinction to make and worthy of research, doesn’t it? We should distill the process to its essence, break down the pieces, remove complexity and dependencies until we understand the utility and effectiveness of all the parts, shouldn’t we?
Possibly relevant for this paper is also the fact that Nerfs are less than 2 years old, while SDFs in rendering are almost 30 years old, and maybe in practice more than that.
Can handle view dependence, can handle transparency, probably faster to train (this paper doesn't mention training speed while the Nvidia one makes a big point about performance), probably higher resolution (comparing final outputs)
It’s important to address the question of what, exactly, is the difference between whatever two things you’re comparing here.
There’s little reason to believe that optimizing an SDF and training a NeRF are any different in terms of optimization speed or resolution, those two processes are really more like different words used to describe the same thing. Training a neural network is an optimization. And NGPs aren’t just a neural network - it also has an explicit field representation.
At this point, neural fields aren’t well defined. The Berkeley NeRFs and Nvidia NGPs are two different things in terms of what the NN lears to infer - one is density the other is SDF. And Yes, these two NN papers are learning material properties in addition to the volumetric representation, while the paper here is learning purely an SDF. That’s simply asking a different question, it’s not a matter of better or worse. The advantages depend on your goals. If all you want is the geometry, then material properties aren’t an advantage, and could add unnecessary complication, right?
> A trainable SDF representation could very well be slower to train than a trainable NeRF representation
Sure. It could very well be faster too (or the same-ish, if it turns out they’re more fundamentally the same than different). Carrying view dependent data around is more bandwidth, potentially significantly more, depending on how you model it. How you model it is under development, and a critical part of the question here.
This all depends on a whole bunch of details that are not settled and can have many implementations. There is significant overlap between the ideas in the Nvidia paper and the EPFL paper, and it’s worth being a bit more careful about defining exactly what it is we’re talking about. It’s easy to say one might be faster. It’s harder to identify the core concepts and talk about what properties are intrinsic to these ideas over a wide variety of implementations, and how they actually differ at their core.
Too true. For example if NeRF has any advantage to this application I will be pleasantly surprised [1] [Text rendering using multi channel signed distance fields]
This reminds me of phase-problem from protein crystallograpy (PX). I'm 15+ years away from that space, so I have no idea what the modern solution space looks like, but it seems like there's a corollary between reverse-AD-over-SDFs (in this work) and the MLE methods for inverse space in PX. It feels like we should be able to take an initial estimate of some SDF over the direct space (2d object) as an ellipsoid, flip to inverse space, do an MLE for phase improvement, and then just do the regular ping-pong on that. The MLE (and successor?) methods are really robust.
Very cool. Presumably the advantage of this is that it can solve big flat areas like the back of the chair that traditional methods never work well with.
But am I understanding correctly that it needs known lighting conditions? Presumably that's why they don't demo it on real images...
What is the problem they are solving? My best guess is: A 3d model is rendered and overlaid a number of background images. From these compositions, reconstruct the original model. Is that it?
I am fairly excited about these techniques. I consider SDFs to be a very powerful representation of 2D scenes, and of course Inigo Quilez has been demonstrating the incredible power of SDFs in 3D. Of course, the big challenge is, what tools do you use to author content in SDF form? In Inigo's case, it's essentially a text editor, coming up with the math yourself, but that doesn't scale unless you're a genius like him.
So using machine learning and optimization techniques in general to solve inverse problems, so your input is something meaningful, is a way to unlock SDF rendering without requiring a reinvention of the whole tool universe.
If someone wants a modest-scope project, it would be applying these kinds of techniques to blurred rounded rectangle rendering[1]. There, I got the math pretty close, fiddling with it by hand, but I strongly suspect it would be possible to get even closer, using real optimization techniques.
Yes, Dreams is also strong evidence that SDF-based rendering is viable. That project also put an enormous amount of effort into creating new design tools :)
I just wish I could use it without learning the gamepad control system. It just doesn't fit my brain and if I step away from it for a few weeks, I've forgotten how to do everything and almost need to start from scratch.
A version of Dreams with either a) a more traditional desktop UI or b) a more intuitive VR-based UI would make me very happy.
I think you were being hyperbolic and I apologize that I must nevertheless offer my opinion that Inigo isn't so much a genius as a craftsman / artist who has been practicing / pioneering in that field. There are many clever ideas he has uncovered but the basics anybody could learn and with practice also create nice visuals by hand.
I mean this is way cooler than nerfs in my opinion. This is just ray tracing (a fairly tractable rendering solution on todays GPUs - whereas nerfs are usually huge on the size and compute fronts) except here the whole pipeline from object to image is differentiable which means you could use it to feed an image in and generate a plausible underlying mesh for it. (Which you can then apply regular physics simulation to or whatever which would have been a pain/impossible with nerfs.)
SDFs do not dictate how you approximate the rendering equation beyond the fact that they are a useful way to describe surfaces. (So, for example, they are mismatched with volumetric rendering.) The characteristics you cite are generally satisfied through the BSSRDF or other material models in conjunction with surface intersection tests, not by raymarching through scattering material. NeRFs solve the entire problem differently by creating a radiance sampling oracle in the form of a neural network. So in other words, yes, you can certainly use SDFs to render those physical effects you mention. Claiming these are comparable to NeRF is pretty misleading, anyway, because NeRF is modeling a scene in a fundamentally different way than surface + material representations, with (other) distinct tradeoffs between the two methods.
It's funny how this totally could have been implemented in like the late 90s except I guess no one did and yet we now see how it's totally able to solve lots of 'inverse problems' that 3d reconstruction algorithms have tried to solve for decades.
One major use case for "convert a bunch of photos into a model" is 3D printing. Often you have an object -- maybe a part which broke and needs to be replaced, maybe a tooth surface which you're making a crown for -- and you need to turn the object into a model and then create a new object.
I see it as a (potential) storage format. As we move away from triangles and towards ray/path/etc tracing (in whatever forms) we want to still have content (be it captured or crafted) in it, that we can render with things like "infinite" details, bounces, and other meta (reflectancr etc).
All current forms of triangulation/meshing from fractcals, sdf, point clouds, etc, are relatively terrible and constrained, to fit existing pipelines (hence why most AR is just lackluster model viewers)
Make it differentiable is a step towards cramming it into a small model and output other forms, or trace directly, or just compress before say, extracting to some Tree-renderable format on the gpu
I did see that in the last NERF paper I scanned. The final representation was smaller than the sparse inputs by a large factor. It got me to thinking along the same lines that this could be like the jpeg/png for 3D scenes online.
There's an incredibly interesting track I've been following trying to find a good latent embedding for AI to generate 3d objects, and SDFs seem like a very good representational format for this. This would, among other things, allow for a DALL-E 2 like generation framework for 3d. Google DeepSDF.
There are other things it's been used for like optimising roughening/regularisation parameters for light transport in order to provide better caustics with forwards path tracing:
I have a problem where I need to do surface/volume estimation of a bunch of objects, but they are moving across the camera with a static background. Anybody know how to do that in real time?
I wonder if anyone is working on combining these techniques with traditional structured light field or lidar geometry capture pipelines. Seems like a fantastic way of both speeding up and improving capture quality.
45 comments
[ 2.6 ms ] story [ 107 ms ] threadNeRFs a are also direction-dependent so work with specular surfaces too.
[1]: https://nvlabs.github.io/instant-ngp/assets/mueller2022insta...
(I edited my post so maybe you replied to an earlier version)
Possibly relevant for this paper is also the fact that Nerfs are less than 2 years old, while SDFs in rendering are almost 30 years old, and maybe in practice more than that.
Can handle view dependence, can handle transparency, probably faster to train (this paper doesn't mention training speed while the Nvidia one makes a big point about performance), probably higher resolution (comparing final outputs)
There’s little reason to believe that optimizing an SDF and training a NeRF are any different in terms of optimization speed or resolution, those two processes are really more like different words used to describe the same thing. Training a neural network is an optimization. And NGPs aren’t just a neural network - it also has an explicit field representation.
At this point, neural fields aren’t well defined. The Berkeley NeRFs and Nvidia NGPs are two different things in terms of what the NN lears to infer - one is density the other is SDF. And Yes, these two NN papers are learning material properties in addition to the volumetric representation, while the paper here is learning purely an SDF. That’s simply asking a different question, it’s not a matter of better or worse. The advantages depend on your goals. If all you want is the geometry, then material properties aren’t an advantage, and could add unnecessary complication, right?
A trainable SDF representation could very well be slower to train than a trainable NeRF representation
> If all you want is the geometry, then material properties aren’t an advantage, and could add unnecessary complication, right?
Unless the SDF's inability to model view dependence would interfere with its ability to minimise its loss
Sure. It could very well be faster too (or the same-ish, if it turns out they’re more fundamentally the same than different). Carrying view dependent data around is more bandwidth, potentially significantly more, depending on how you model it. How you model it is under development, and a critical part of the question here.
This all depends on a whole bunch of details that are not settled and can have many implementations. There is significant overlap between the ideas in the Nvidia paper and the EPFL paper, and it’s worth being a bit more careful about defining exactly what it is we’re talking about. It’s easy to say one might be faster. It’s harder to identify the core concepts and talk about what properties are intrinsic to these ideas over a wide variety of implementations, and how they actually differ at their core.
Too true. For example if NeRF has any advantage to this application I will be pleasantly surprised [1] [Text rendering using multi channel signed distance fields]
[1] https://news.ycombinator.com/item?id=20020664
- SDFs are much more amenable to direct manipulation by humans
- SDFs potentially can be decomposed into more human-understandable components
- SDFs may provide a better mechanism for constructing a learned latent space for objects (see eg DeepSDF)
- Some rendering engine targets may play more nicely with SDFS
[1] https://pubmed.ncbi.nlm.nih.gov/35362474/
But am I understanding correctly that it needs known lighting conditions? Presumably that's why they don't demo it on real images...
IF true THEN ‘big’ END IF
So using machine learning and optimization techniques in general to solve inverse problems, so your input is something meaningful, is a way to unlock SDF rendering without requiring a reinvention of the whole tool universe.
If someone wants a modest-scope project, it would be applying these kinds of techniques to blurred rounded rectangle rendering[1]. There, I got the math pretty close, fiddling with it by hand, but I strongly suspect it would be possible to get even closer, using real optimization techniques.
[1] https://raphlinus.github.io/graphics/2020/04/21/blurred-roun...
I do think SDFs are a good path for a great potential breakthrough for AI directed 3d modelling tools like DALL-E -- we're pretty close.
A version of Dreams with either a) a more traditional desktop UI or b) a more intuitive VR-based UI would make me very happy.
I agree with the rest of what you are saying.
Does anyone know what the end goal of this kind of research is or why there is so much interest?
Its definitely cool but is the idea just to make photogrammetry cheaper/easier?
Or are there other use cases I'm missing
All current forms of triangulation/meshing from fractcals, sdf, point clouds, etc, are relatively terrible and constrained, to fit existing pipelines (hence why most AR is just lackluster model viewers)
Make it differentiable is a step towards cramming it into a small model and output other forms, or trace directly, or just compress before say, extracting to some Tree-renderable format on the gpu
https://diglib.eg.org/handle/10.1111/cgf14347