Show HN: I've made a Monte-Carlo raytracer for glTF scenes in WebGPU (github.com)
This is a GPU "software" raytracer (i.e. using manual ray-scene intersections and not RTX) written using the WebGPU API that renders glTF scenes. It supports many materials, textures, material & normal mapping, and heavily relies on multiple importance sampling to speed up convergence.
42 comments
[ 3.7 ms ] story [ 99.7 ms ] threadHow far have we gotten in terms of training AI models on raytraced lighting, to simulate it but fast enough for video games? Training an AI not on rendered scenes from any particular viewpoint, but rather on how light and shadows would be "baked into" textures?
Because what raytracing excels at is the overall realism of diffuse light. And it seems like the kind of thing AI would be good at learning?
I've always though, e.g. when looking at the shadows trees cast, I couldn't care less if the each leaf shape in the shadow is accurate or entirely hallucinated. The important things seem to be a combination of the overall light diffusion, combined with correct nearby shadow shapes for objects. Which is seems AI would excel at?
Nvidia's DLSS is a neural network that upscales images so that games may be rendered quickly at lower resolutions, and than upscaled to the display resolution in less total time than rendering natively at the display resolution.
Nvidia's DLDSR downscales a greater-than-native resolution image faster than typical downscaling algorithms used in DSR.
Nvidia's RTX HDR is a post-processing filter that takes an sRGB image and converts it to HDR.
So, it is very likely that a model that converts rasterized images to raytraced versions is possible, and fast. The most likely road block is the lack of a quality dataset for training such a model. Not all games have ray tracing, and even fewer have quality implementations.
How would this even work and not just be a DLSS derivative?
The magic of ray tracing is the ability to render light sources and reflections that are not in the scene. So where is the information coming from that the algorithm would use to place and draw the lights, shadows, reflections, etc?
I'm not asking to be snarky. I can usually "get there from here" when it comes to theoretical technology, but I can't work out how a raster image would contain enough data to allow for accurate ray tracing to be applied for objects whose effects are only included due to ray tracing.
The ML part of DLSS is that the blend weights are determined by a neural net, rather than handwritten heuristics.
DLSS 1 _did_ try and and use neural networks to predict the new (upscaled) pixels outright, which went really poorly for a variety of reasons I don't feel like getting into, hence why they abandoned that approach.
1) commercial hardware pipelinea being improved for decades in handling 3D polygons, and
2) graphical AI models are trained on understanding natural language in addition to rendering.
I can imagine a new breed of specialized generative graphical AI that entirely skips language and is trained on stock 3D objects as input, which could potentially perform much better.
The main benefit of raytracing in games is that it has accurate global knowledge of the scene beyond what's directly in front of the camera, as opposed to earlier approximations which tried to work with only what the camera sees. Img2img diffusion is the ultimate form of the latter approach in that it tries to infer everything from what the camera sees, and guesses the rest.
Also, that new Google paper where it generates entire games from a single image has up to 60 seconds of 'memory' I think they said, so I don't think the "forgetting" is actually that big of a problem since we can refresh the memory with a properly rendered image at least every that often.
I'm just spitballing here though, I think all of Unreal 5.4 or 5.5 has put this into practice already with their new lighting system.
That's already a thing, there's ML-driven denoisers which take a rough raytraced image and do their best to infer what the fully converged image would look like based on their training data. For example in the offline rendering world there's Nvidia's OptiX denoiser and Intel's OIDN, and in the realtime world there's Nvidia's DLSS Ray Reconstruction which uses an ML model to do both upscaling and denoising at the same time.
https://developer.nvidia.com/optix-denoiser
https://www.openimagedenoise.org
Specifically this one, which seems to tackle what you mentioned: https://research.nvidia.com/labs/rtr/publication/hadadan2023...
Even with the fact that it's static lighting, you can already see a ton of the challenges that they faced. In the end they did get a fairly usable solution that improved on their existing baking tools, but it took what seems like months of experimenting without clear linear progress. They could have just as easily stalled out and been stuck with models that didn't work.
And that was just for static lighting, not every realtime dynamic lighting. ML is going to need a lot of advancements before it can predict lighting whole-sale, faster and easier than tracing rays.
On the other hand ML is really really good at replacing all the mediocre handwritten heuristics 3d rendering has. For lighting, denoising low-signal (0.5-1 rays per pixel) lighting is a big area of research[0] since handwritten heuristics tend to struggle with such little amount of data available, along with lighting caches[1] which have to adapt to a wide variety of situations that again make handwritten heuristics struggle.
[0]: https://gpuopen.com/learn/neural_supersampling_and_denoising..., and the references it lists
[1]:https://research.nvidia.com/publication/2021-06_real-time-ne...
> // No idea where negative values come from :(
I don't know, but:
> newRay.origin += sign(dot(newRay.direction, geometryNormal)) * geometryNormal * 1e-4;
The new origin should be along the reflected ray, not along the direction of the normal. This line basically adds the normal (with a sign) to the origin (intersection point), which seems odd.
Poor's man way to find where the negatives come from is to max(0,...) stuff until you find it.
Sure! If you look into the to-do list, there's a "wavefront path tracer" entry :)
> new origin should be along the reflected ray
I've found that doing it the way I'm doing it works better for preventing self-intersections. Might be worth investigating, though.
Do we have good sorting strategy whose costs are amortized yet? Meister 2020 (https://meistdan.github.io/publications/raysorting/paper.pdf) shows that the hard part is actually to hide the cost of the sorting.
> squeeze in an adaptive sampler in the middle. Can you expand on that? How does that work? I only know of adaptive sampling in screen space where you shoot more or less rays to certain pixels based on their estimated variance so far.
They also do talk about the potential of ray reordering for complex scenes and complex materials in the paper (because reordering helps with shading divergence since "all" reordered rays are pretty much going to hit the same material).
So maybe ray reordering isn't dead just yet. Probably would have to try that at some point...
On the web, WebGPU is only supported by Chrome-based browser engines at this point, and a lot of software developers us Firefox (and don't really like encouraging a browser monoculture), so it doesn't make a ton of sense to target browser based WebGPU for some people at this point.
> WebGPU
> this project is desktop-only
Boss, I am confused, boss.
My browser doesn't support WebGPU properly yet, so I don't really care about running this thing in browser.
And it gets me a bit sad about the state of WebGPU, however hopefully that'll be resolved soon... I also on Linux am impatiently waiting for WebGPU to be supported on my browser.