Those interested in this article may be interested in experimenting with the Mitsuba renderer [1]. This is one of the best open-source spectral renderers, and it includes support for RGB and spectral modes (in addition to propagating things like polarization), all in one unified rendering pipeline with CUDA and non-CUDA backends.
It seems strange that the website of a renderer wouldn’t have any rendered images. What does it look like? Do spectral renderers achieve a better result than traditional rendering methods?
But Mitsuba is not really complete system to make pretty pictures, it is more of a framework for researchers to prototype algorithms etc. So while many state of the art methods have been implemented on top of Mitsuba, not nearly all of them have been merged to the mainline. If you want comparisons then you probably need to look up specific papers to see comparisons for the specific problems they solve.
Better is subjective, but the whole point of spectral rendering is to be more objective. The visual difference is minor and its relevance to film/games is put into question by the artistic post processing that inevitably distorts the final result. There are some effects that “just work” with spectral rendering though, so you might save some hackery there.
Another stand-out feature of Mitsuba is its focus on differentiability. Wenzel Jakob is an absolute rockstar in computer graphics, and his group creates informative demo and explanatory videos for their areas of work. These include:
Also worth mentioning Manuka, which is Weta's production spectral renderer. There's a nice overview of the architecture here, which includes a section on their motivations for, and implementation of, spectral light transport:
OP author here. Thanks for mentioning it, I happen to work at Wētā FX and we are obviously very biased toward spectral rendering and the problems it solves. I often hear people saying that spectral rendering is complicated but it is actually the opposite, mostly everything becomes simplified with it. It, gives, at the very least an elegant first principle based framework to work from.
It is baffling just how well RGB rendering works despite it being such a huge hack; many real-world scenes are practically indistinguishable between rgb and spectral rendering. But still I find the physically based rendering pretty disingenuous considering the huge gap still remaining between real-world and standard pbr although I accept that it is just a domain-specific term that shouldn't be taken too literally.
> I find the physically based rendering pretty disingenuous considering the huge gap still remaining between real-world and standard pbr
> ... I accept that it is just a domain-specific term that shouldn't be taken too literally.
Actually it is supposed to be taken literally. That's why it's physically based rendering and not physically correct rendering. The goal is to create a system based on real world physics to get as close as possible within the constraints of current computer hardware.
I believe this article gets it wrong:
> The primary goal of physically-based rendering (PBR) is to create a simulation that accurately reproduces the imaging process of electro-magnetic spectrum radiation incident to an observer. This simulation should be indistinguishable from reality for a similar observer.
I don't think the stated goal of PBR is to be indistinguishable from reality for a similar observer. At least, I have never seen this stated anywhere.
However, it's worth noting that PBR is a loaded term and includes both physically based lighting, and physically based shading (surface properties). If we are purely talking about lighting, that is, modelling the way light bounces in a physical way, it might be possible to get an exact match to physical reality. Light bounces in a relatively simple way.
The problem is that surfaces are not simple. They absorb, reflect, diffract, scatter, and so on. So even if the equations modelling the reaction of light are 100% physically correct, it's an entirely different story to model a complex surface like skin in a physically correct way.
This article is about light, so it's talking about physically based lighting. In which case the phrase this simulation should be indistinguishable from reality for a similar observer is more reasonable, but I think it should still be qualified for clarity.
Indeed. The book[1] on the subject states that the point is that, unlike the rendering methods that came before, the math is based on physics and physical principles. Of course one has to simplify and approximate, but the approximations should be grounded in physics and obey the underlying physical principles.
For example, in the early days people created reflection models without considering energy reciprocity for example. Well that's a physical principle that's very well established in physics[2] so any physically based renderer should only have reflection models that obey that principle.
In a sense all (non-stylized) rendering is physics-based in the sense that all that physics is a model, and indeed the field of physics has different models of differing accuracy/precision/applicability. So even pre-PBR could be called physical in the sense that it represents some sort model of light, the rest is more of question on degree of accuracy and applicability. What I'm aiming at is that the line drawn for PBR is drawn in sand and feels somewhat arbitrary.
It's not that arbitrary to me. The earlier work was sometimes physically-motivated, and sometimes even based on physics. For example using Lambertian reflection[1] for diffuse surfaces.
But the key, in my view, to PBR is that the whole pipeline is based on physical principles and math, while the old renderers were not. So previous renderers would not bother properly normalizing the Lambertian coeffients for example, leading to violation of energy conservation.
> I don't think the stated goal of PBR is to be indistinguishable from reality for a similar observer. At least, I have never seen this stated anywhere.
I’m, admittedly, highly biased as I work for Wētā FX, a high-end visual effects vendor. We have been developing for over a decade a spectral renderer, i.e. Manuka [1], whose design principles and philosophy are aimed at recreating how a motion picture camera, its lenses, filters, and even colour science, image the real world so that we can integrate virtual cg elements into plates as if they were part of the original photography. This hopefully gives some context to your quote. Whether we are successful at this is a different topic but this is a core business offering we are hired for.
Hey, thanks for writing the article! My nitpicks aside, I learned a lot. I'm always interested in reading more about spectral rendering.
> I’m, admittedly, highly biased as I work for Wētā FX, a high-end visual effects vendor
That's fair. I can see it being a stated goal of PBR at Weta. However, I'm at the extreme other end of the rendering spectrum. I work with WebGL (I contribute a lot to three.js) and being indistinguishable from reality is most assuredly NOT a goal of a real time in browser rendering engine. I mean, we can dream. But it will be a long, long time before we could realize that dream so our time is better spent finding approximations that can work even on a midrange mobile device.
Is it that baffling? After all, the core principle is that instead of representing a continuous spectra, you project the spectra to a few basis functions[1], so you can later reconstruct an approximation of the original spectra.
Our eyes have three different cone cells with different responses[2], so the fact that three basis functions[3] that roughly align with the cone cells work well as an approximation isn't that surprising to me.
That said, way back when I was doing rendering I was toying with the idea of trying to find a optimal basis functions for rendering, like if scene-specific ones could improve things.
For basic direct lighting its not all that weird, but for any indirect lighting or translucency etc it is pretty whacky. I guess it basically boils down to most real-world scenes having both fairly broadband light-sources and typical material spectrum being also relatively broadband, and in general having smooth spectrums. I'd imagine if real-world for some reason predominantly consisted of super spiky or notchy spectrums then the RGB rendering would struggle more. Or maybe not.
Yes, indeed. If you have narrow-band light sources, say a mercury vapor lamp[1], and some material that only reflects a different narrow-band spectrum, then you should get very little reflection while with an RGB renderer you'll get some reflection.
However as you point out, very few materials and lights are like this, hence why the approximation works well.
yes, I suppose the 'baffling' part is partially from that theoretically there are infinitely (or nearly so) many different spectrums that produce some one specific color perception. So sort of probabilistically it could be seen as lucky that the spectrums of lights and materials align in a way that they do in real world.
I wonder what are the pathological cases here. I'm imagining some setup where an object could appear variably green, yellow, or red; this could be maybe accomplished by different mixes of different red/green lasers and maybe white light. The paint might be bigger problem :)
The eye also does a lot of work here, see color constancy[1]. BBC had a nice documentary on this back in the days, Colorful Notions[2], see the 19min mark for an experiment showing this in action.
I think translucency is mostly whacky because it's typically represented by a single alpha channel, and a tint is represented by a color added on top. Which is not how most transparent materials look like.
I think we could do better, even in RGB by treating translucency as a 3x3 matrix instead of a single alpha scalar. Per-channel transparency could be represented by a diagonal matrix, but there is no reason that the current RGB basis would be adequate for representing transparent materials, hence the generalization to a 3x3 matrix.
Then a tint wouldn't need to be defined additively, a green tinted glass could simply not let most of red and blue through.
Tristimulus values (X, Y, Z) are the dot product of the incoming spectral radiance (L) and the spectral sensitivity curves (x, y, z) of the three cone cell types:
X = dot(L, x)
Y = dot(L, y)
Z = dot(L, z)
The spectrum of the light reflected from a surface (Lᵣ) is the pointwise product of the incoming spectral radiance (L) and the spectral reflectance curve of the surface (R):
Lᵣ = L * R
What RGB rendering does, is it replaces this equation (for x, y, z):
X = dot(Lᵣ, x) = dot(L * R, x) (eq I)
with this (for x, y, z):
X = dot(L, x) * dot(R, x) (eq II)
Which are not equal in the general case. On the other hand, they are equal up to a scaling factor if L is a constant function, since in that case:
where C and D are scaling factors that depend solely on L. The interpretation of this is that as long as the light source has a sufficiently flat spectrum (like sunlight!), the difference between spectral rendering and RGB rendering will be negligible.
> as long as the light source has a sufficiently flat spectrum (like sunlight!), the difference between spectral rendering and RGB rendering will be negligible.
(Great comment, nitpick ahead)
...to the human eye, that is. For industrial applications spectral rendering can still be useful.
one caveat is that (AFAIK) if you are doing optical simulations in your rendering, like prisms or glass, the spectral rendering would produce more accurate caustics/reflections/refractions. But to be fair, most realtime rendering is not doing that. I could imagine it being important for stuff like CG films.
This is correct, however, not everything is lit by sunlight, especially onset in the motion picture industry, where solid state lighting and its narrow band irradiance sources, e.g. led lights or led walls, are prevalent. Those emission sources induce a lot of unexpected appearance problems, e.g. metameric failures that RGB rendering cannot possibly solve. Our spectral renderer, i.e. Manuka [1], was specifically designed to address those issues whilst imaging with a virtual motion picture camera.
One recent example I have in mind is The Batman (2022): There are sequences/shots lit with high-pressure sodium lights (HPS), those have a flat spectrum with spikes between 500 and 600nm, the appearance of objects lit under them is very specific and can only be accurately reproduced with a spectral renderer such as Manuka, Mitsuba [2] or Art [3].
100% agreed! By all means, I didn't want to dismiss spectral rendering. I just wanted to give some context and justification as to why RGB rendering looks plausible in most scenarios.
I've been working with a hyperspectral camera (350-1000nm,4nm,12b,410x410,8Hz) lately and this article was quite interesting through that specific lens. I'm pretty much in love with hyperspectral imaging now and it's really opened my eyes to a whole new world of information and technology. Not super relevant, just thought I'd share. Also, this camera has really made me want a real-life tricorder.
You might enjoy the ECOSTRESS Spectral Library [1]. It has downloadable reflectance spectra overlapping with your sensor for a variety of materials. These spectra are collected in highly controlled lab conditions for pure material samples. Could be useful for calibration or making your tricorder ;)
I do consulting work in this area, in case you've ever got questions or could use a hand developing something.
31 comments
[ 3.2 ms ] story [ 51.4 ms ] thread[1]: https://www.mitsuba-renderer.org/
Show me! I want to see the difference!
But Mitsuba is not really complete system to make pretty pictures, it is more of a framework for researchers to prototype algorithms etc. So while many state of the art methods have been implemented on top of Mitsuba, not nearly all of them have been merged to the mainline. If you want comparisons then you probably need to look up specific papers to see comparisons for the specific problems they solve.
Differentiable rendering: http://rgl.epfl.ch/publications/Jakob2022DrJit
Volumetric rendering: http://rgl.epfl.ch/publications/NimierDavid2022Unbiased
Optimization of light path choice via AD: http://rgl.epfl.ch/publications/Vicini2021PathReplay
https://jo.dreggn.org/home/2018_manuka.pdf
I played my part in this back in the 2010s maintaining the blender integration, fun times :)
But both the renderer and the integrations got pretty much entirely re-written in the move to GPU compute shortly after that time.
Just to spark discussion, here is one example of research aiming to reduce that gap and bring more physicality: https://ssteinberg.xyz/2023/03/27/rtplt/
> ... I accept that it is just a domain-specific term that shouldn't be taken too literally.
Actually it is supposed to be taken literally. That's why it's physically based rendering and not physically correct rendering. The goal is to create a system based on real world physics to get as close as possible within the constraints of current computer hardware.
I believe this article gets it wrong:
> The primary goal of physically-based rendering (PBR) is to create a simulation that accurately reproduces the imaging process of electro-magnetic spectrum radiation incident to an observer. This simulation should be indistinguishable from reality for a similar observer.
I don't think the stated goal of PBR is to be indistinguishable from reality for a similar observer. At least, I have never seen this stated anywhere.
However, it's worth noting that PBR is a loaded term and includes both physically based lighting, and physically based shading (surface properties). If we are purely talking about lighting, that is, modelling the way light bounces in a physical way, it might be possible to get an exact match to physical reality. Light bounces in a relatively simple way.
The problem is that surfaces are not simple. They absorb, reflect, diffract, scatter, and so on. So even if the equations modelling the reaction of light are 100% physically correct, it's an entirely different story to model a complex surface like skin in a physically correct way.
This article is about light, so it's talking about physically based lighting. In which case the phrase this simulation should be indistinguishable from reality for a similar observer is more reasonable, but I think it should still be qualified for clarity.
For example, in the early days people created reflection models without considering energy reciprocity for example. Well that's a physical principle that's very well established in physics[2] so any physically based renderer should only have reflection models that obey that principle.
[1]: https://www.pbrt.org/
[2]: https://en.wikipedia.org/wiki/Helmholtz_reciprocity
But the key, in my view, to PBR is that the whole pipeline is based on physical principles and math, while the old renderers were not. So previous renderers would not bother properly normalizing the Lambertian coeffients for example, leading to violation of energy conservation.
[1]: https://en.wikipedia.org/wiki/Lambertian_reflectance
> I don't think the stated goal of PBR is to be indistinguishable from reality for a similar observer. At least, I have never seen this stated anywhere.
I’m, admittedly, highly biased as I work for Wētā FX, a high-end visual effects vendor. We have been developing for over a decade a spectral renderer, i.e. Manuka [1], whose design principles and philosophy are aimed at recreating how a motion picture camera, its lenses, filters, and even colour science, image the real world so that we can integrate virtual cg elements into plates as if they were part of the original photography. This hopefully gives some context to your quote. Whether we are successful at this is a different topic but this is a core business offering we are hired for.
- [1] https://www.wetafx.co.nz/research-and-tech/technology/manuka...
> I’m, admittedly, highly biased as I work for Wētā FX, a high-end visual effects vendor
That's fair. I can see it being a stated goal of PBR at Weta. However, I'm at the extreme other end of the rendering spectrum. I work with WebGL (I contribute a lot to three.js) and being indistinguishable from reality is most assuredly NOT a goal of a real time in browser rendering engine. I mean, we can dream. But it will be a long, long time before we could realize that dream so our time is better spent finding approximations that can work even on a midrange mobile device.
Our eyes have three different cone cells with different responses[2], so the fact that three basis functions[3] that roughly align with the cone cells work well as an approximation isn't that surprising to me.
That said, way back when I was doing rendering I was toying with the idea of trying to find a optimal basis functions for rendering, like if scene-specific ones could improve things.
[1]: https://www.psych.mcgill.ca/misc/fda/ex-basis-a1.html
[2]: https://en.wikipedia.org/wiki/Cone_cell
[3]: https://en.wikipedia.org/wiki/CIE_1931_color_space
However as you point out, very few materials and lights are like this, hence why the approximation works well.
[1]: https://en.wikipedia.org/wiki/Mercury-vapor_lamp
I wonder what are the pathological cases here. I'm imagining some setup where an object could appear variably green, yellow, or red; this could be maybe accomplished by different mixes of different red/green lasers and maybe white light. The paint might be bigger problem :)
[1]: https://en.wikipedia.org/wiki/Color_constancy
[2]: https://vimeo.com/11932120 (sadly poor quality, better rips can be found)
I think we could do better, even in RGB by treating translucency as a 3x3 matrix instead of a single alpha scalar. Per-channel transparency could be represented by a diagonal matrix, but there is no reason that the current RGB basis would be adequate for representing transparent materials, hence the generalization to a 3x3 matrix.
Then a tint wouldn't need to be defined additively, a green tinted glass could simply not let most of red and blue through.
(Great comment, nitpick ahead)
...to the human eye, that is. For industrial applications spectral rendering can still be useful.
This is correct, however, not everything is lit by sunlight, especially onset in the motion picture industry, where solid state lighting and its narrow band irradiance sources, e.g. led lights or led walls, are prevalent. Those emission sources induce a lot of unexpected appearance problems, e.g. metameric failures that RGB rendering cannot possibly solve. Our spectral renderer, i.e. Manuka [1], was specifically designed to address those issues whilst imaging with a virtual motion picture camera.
One recent example I have in mind is The Batman (2022): There are sequences/shots lit with high-pressure sodium lights (HPS), those have a flat spectrum with spikes between 500 and 600nm, the appearance of objects lit under them is very specific and can only be accurately reproduced with a spectral renderer such as Manuka, Mitsuba [2] or Art [3].
- [1] https://www.wetafx.co.nz/research-and-tech/technology/manuka...
- [2] https://www.mitsuba-renderer.org/
- [3] https://cgg.mff.cuni.cz/ART/about/
I do consulting work in this area, in case you've ever got questions or could use a hand developing something.
[1]: https://speclib.jpl.nasa.gov/
I'll certainly keep you in mind if I need/want some help in the future.