Do note that the following is part of the article and it's from 2018, not sure about it's accuracy anymore: "As denoted by the ‘NVX’ prefix, this API is not yet final and could undergo some minor changes before the final release."
Can anyone give me any intuitive understanding of how ray tracing can be adapted into a conventional rasteriser workflow?
Doesn't ray tracing need a persistent, whole-world scene graph? Rays can end up anywhere, no matter where the camera is pointing, isn't that right? I thought the whole scene wasn't on the GPU? I thought the CPU streamed polygons to be drawn based on what was likely to be visible, but it can't stream the entire world every draw can it?
The data for each scene hasn't been streamed to the GPU in a long time, not since the first few generations of GeForce cards. "Vertex buffers" store the geometry in the GPU's RAM, so it's there. The game engine just swaps out buffers as things come in and out of view.
I have the same questions as you regarding how this works with respect to offloading the the scene graph data structures though.
To shoot rays in scene you create acceleration structure (internal to driver/GPU hardware). Input to this structure are vertex & index buffers & model transform matrix. Geometry is specified as triangles or AABB. You can update this structure later by providing new transform matrix. There are different flags that control performance/memory usage - fast tracing or updates vs less memory usage.
Yes. Driver/GPU handles that. It "just works". You just manage things on high level - what can be moved in world, what cannot and group the updates together.
> Rays can end up anywhere, no matter where the camera is pointing, isn't that right?
Only indirect or secondary rays. Primary rays (the ones from the camera) can only hit things that are visible to the camera.
Also, ray-tracing trivially allows for breaking up the rendering in multiple steps.
You could for example have a pre-made acceleration structure for the static part of the world, a less efficient but fast-building one for dynamic objects in the world, cast against either and see which intersection has the shortest distance. This could be done in parallel, and you can be more clever about it too depending on circumstances.
Not saying that's what people do, I've been out of the game for too long, but at least it's an option.
I think you're generally correct, but that games studios and animators don't care much about refraction specifically (although it does make for cool tech demos). They care more about reflection and lighting in general (spotlights such as windows, true area lights - no point lights disguised as area lights using emmisive textures)
Ray tracing can be used for different camera projections, sharp reflections, sharp refractions, sharp shadows and other things like physics. (Also it's rasterize)
People keep giving you the answers and you don't seem to accept them. You said you thought it was just useful for refraction, this is what it is useful for in games, including different camera projections.
Another person replied and told you that the acceleration structure can be dynamic with geometry inserted and taken out.
You said you thought ray tracing was never used for direct rays and only for indirect days along with asking how the rays have access to the full scene geometry and both questions have been answered thoroughly.
> You said you thought ray tracing was never used for direct rays
I didn't say this anywhere. You're imagining extra words between what I actually said. Everyone else in this thread understood my question, and they managed to answer it without also ignorantly trying to correct my spelling.
You were told that rays from the camera would only hit what is visible to the camera.
Also you might want to get upset at Nvidia, AMD, Pixar and every book on computer graphics if getting offended over spelling what you want to focus on.
>>But what is visible to the camera from rasterisation is different to what is visible to a camera using ray tracing! That was the point!
That is not true.
>>You were the one who first got offended by spelling and tried to correct me to do it your way!
I wasn't offended, I just told you the way that it's spelled by the vast majority of people who invented and work with these things. It isn't 'my' way, I'm not sure why you are so hung up on this. Maybe you are trying to avoid answers you don't like to the incorrect assumptions in your questions.
No it is true - a polyglon not visible in rasterisation could become visible in ray tracing if the ray passes through a refractive material. So you may need additional geometry. Get it?
Those are secondary rays, which are separate from the primary rays from a camera. Secondary rays from reflective or transmissive effects are not considered visibility from the camera. This distinction was very important in hybrid renderers that combined rasterization from the camera with ray tracing for secondary rays, like PRman and mental ray. Now we have the same patterns being repeated in real time rendering. This was explained to you already. You could learn something if you didn't think you knew everything already.
One of the turning points was to actually work backwards: instead of calculating every ray from a light source, then throwing away anything that doesn’t hit the camera, ray tracing now starts with the ray endpoints that end at the camera, thus saving all that work that needs to be thrown out the other way.
Reflections do need access to the geometry off-camera, but it’s accessible in some for or another by needing to be available when the player looks at it, as well as needing to exist for gameplay interaction (such as an enemy shooting from off-camera)
Actually once you get a sufficient number of triangles, it can be faster to ray-trace than rasterize, as rasterized triangles have to be drawn, while ray-traced triangles can be skipped thanks to the acceleration structure.
Anyway, yeah sure the fun is typically doing secondary rays, but you could do those with the tricks I mentioned, and/or with reduced detail (thus faster to update) unless you're doing perfect reflection/refraction.
On top of what other people wrote, I think a common misconception is that raytracing is only useful to determine the final shaded color of a pixel on the screen. In its current form, things like RTX are still way too slow to do this at any appreciable quality level.
You can use raytracing passes at very different resolutions, quality settings, geometry detail levels etc though, and combine them with the 'regular' render pipeline later. For example to calculate environment maps for reflections, shadow maps, or even things completely unrelated to drawing (hit tests, sound reflection etc). Suppose I want to have a nice reflection in a pool of water for example, you can already get a very good looking effect by raytracing a low-resolution environment map from the water pools' perspective, which wouldn't need any tracing of secondary rays etc, and use it in your water shader.
There's uncountable ways something like this could be useful (not all of which can be accelerated very well though, I guess)
For doing the more traditional raytracing at any resolution you will always need an acceleration structure like a BVH though, but building/updating that is part of these kinds of hardware extensions.
Nice to see Intel, AMD, and Nvidia working together on an open standard. Do Apple's in-house GPUs have Vulkan support or would something like this have to go through a Vulkan-on-Metal translation layer instead?
Looking more and more like Apple will never touch Vulkan directly. They are heavily invested in cross-device performance graphics using Metal.
The more the API situation evolves, I can see Google, Microsoft and Sony going all in all Vulkan for performance graphics. Only if MS is willing to part ways with D3D12
For cross platform I could see WebGPU + WebAssembly being the choice, where it abstracts both Metal and Vulkan.
Honestly I feel like D3D12's days are numbered. Tons of engines and devs are moving to Vulkan and while Vulkan has tons of tutorials and documentation, D12 has NOTHING. If you want to just go out and make a D12 game, you basically need to join up with one of the existing DirectX shops that has all the knowledge siloed within.
Interesting. I haven't been following the games industry for a while. Thanks for the insights! Given Microsoft's move towards more open ecosystems, I can definitely see it happening. Especially launching Android devices at year's end where I am sure cross-device gaming (TV + Mobile) will become important. I know Android is increasingly focused on Vulkan
I disagree. Sony / Nintendo use proprietary APIs, Windows and Xbox use DirectX, and Android you need OpenGL ES for compatibility. Apple is self-disqualifying with their hardware and insistence on Metal.
That leaves Vulkan for Linux games. The 1% of the market that no savvy business person would touch with a 10 feet pole.
In Linux land, you'll meet people who feel suspicious of you in the first place for charging for your game, and people who have compiled their GPU driver from scratch, yet blame your game for being incompatible with their wrong compiler flags.
So for almost every game studio, there is no viable market where you could sell a Vulkan game.
Nintendo and Android use Vulkan.
Windows supports it as well (along opengl and directX) so it's a bit more than just Linux (which powers Stadia and will likely power most Stadia-clones in the future).
This pretty much only leaves out macOS and iOS, for which a Vulkan compatibility layer exists anyways.
I agree they you can use Vulkan on most platforms, but if you use an established engine with multiple render backends, then you'd go with the most feature complete or most performant API, which is usually the platform-specific proprietary one.
That would limit Vulkan as a good cross-platform choice to the few developers that do not use Unity or Unreal Engine under the hood.
Not quite true, although it tends to be used a lot as Vulkan sales pitch.
Nintendo uses Vulkan on the Switch, but they also have their own Metal like API, NVN.
Vulkan is mostly used by PC ports into the Switch.
Exclusive Switch titles do so via NVN, Unreal or Unity bindings to NVN, not Vulkan.
Android has optional support for Vulkan, since only flagship devices ever bothered to actually provide it, with Android 10 Google has changed it into compulsory API. So only Android 10 or later devices are guaranteed to actually have proper Vulkan drivers available.
Windows supports Vulkan on the classic OpenGL ICD driver model, not available in Win32 and UWP sandboxes, which only allow for DirectX based code.
This is the actual reality, and not that typical Vulkan sales pitch.
I like this. I will use the word "sales pitch" next time other companies does it, which to me is flat out lying. ( Cough; AV1 )
I still dont get why the hype on Vulkan, if you want Android support you will still need to use OpenGL for the majority of devices, given their slow replacement cycle and I doubt it will be mainstream anytime soon.
I disagree. Vulkan sounds nice and all, but it's terribly hard to actually use in practice. It has cost me a lot of evenings to compile and run some Vulkan examples. The same thing in OpenGL or DirectX can be done in one evening.
On top of that, Vulkan is much hard to actually use in practice. It's much more low-level than OpenGL and DirectX.
Based on a recent post about WGSL (the WebGPU shading language) it sounds like Apple is in a legal battle with Khronos so Apple seems unlikely to adopt anything Khronos related.
this might be why WebGL2 never really shipped on Safari. They added a flag but there is no actual implementation.
I don't have inside information, however from some occasional stuff I have read here and there, it seems Khronos was unwilling to drive OpenCL the way Apple wished for, and they weren't happy to have lost control over OpenCL.
If this actually the reason, I don't know, corrections welcomed.
46 comments
[ 3.3 ms ] story [ 102 ms ] threadVulkan Ray Tracing: https://is.gd/DMIKVs
Do note that the following is part of the article and it's from 2018, not sure about it's accuracy anymore: "As denoted by the ‘NVX’ prefix, this API is not yet final and could undergo some minor changes before the final release."
Doesn't ray tracing need a persistent, whole-world scene graph? Rays can end up anywhere, no matter where the camera is pointing, isn't that right? I thought the whole scene wasn't on the GPU? I thought the CPU streamed polygons to be drawn based on what was likely to be visible, but it can't stream the entire world every draw can it?
I have the same questions as you regarding how this works with respect to offloading the the scene graph data structures though.
https://devblogs.nvidia.com/practical-real-time-ray-tracing-...
See following links:
* https://www.khronos.org/registry/vulkan/specs/1.2-extensions...
* https://www.khronos.org/registry/vulkan/specs/1.2-extensions...
Only indirect or secondary rays. Primary rays (the ones from the camera) can only hit things that are visible to the camera.
Also, ray-tracing trivially allows for breaking up the rendering in multiple steps.
You could for example have a pre-made acceleration structure for the static part of the world, a less efficient but fast-building one for dynamic objects in the world, cast against either and see which intersection has the shortest distance. This could be done in parallel, and you can be more clever about it too depending on circumstances.
Not saying that's what people do, I've been out of the game for too long, but at least it's an option.
Isn't the whole point of ray tracing things like refraction? Otherwise why not just rasterise?
I know. But you may need geometry that isn’t trivially visible in order to render these things.
> (Also it's rasterize)
I’m British. Not everyone does things the American way. Don’t try to push your approach on everyone else.
Another person replied and told you that the acceleration structure can be dynamic with geometry inserted and taken out.
"things like refraction". Not "just".
I didn't say this anywhere. You're imagining extra words between what I actually said. Everyone else in this thread understood my question, and they managed to answer it without also ignorantly trying to correct my spelling.
Also you might want to get upset at Nvidia, AMD, Pixar and every book on computer graphics if getting offended over spelling what you want to focus on.
(Also spelling is not an approach to something)
But what is visible to the camera from rasterisation is different to what is visible to a camera using ray tracing! That was the point!
> you might want to get upset at Nvidia, AMD, Pixar and every book on computer graphics if getting offended over spelling what you want to focus on
You were the one who first got offended by spelling and tried to correct me to do it your way!
That is not true.
>>You were the one who first got offended by spelling and tried to correct me to do it your way!
I wasn't offended, I just told you the way that it's spelled by the vast majority of people who invented and work with these things. It isn't 'my' way, I'm not sure why you are so hung up on this. Maybe you are trying to avoid answers you don't like to the incorrect assumptions in your questions.
No it is true - a polyglon not visible in rasterisation could become visible in ray tracing if the ray passes through a refractive material. So you may need additional geometry. Get it?
Anyway, yeah sure the fun is typically doing secondary rays, but you could do those with the tricks I mentioned, and/or with reduced detail (thus faster to update) unless you're doing perfect reflection/refraction.
You can use raytracing passes at very different resolutions, quality settings, geometry detail levels etc though, and combine them with the 'regular' render pipeline later. For example to calculate environment maps for reflections, shadow maps, or even things completely unrelated to drawing (hit tests, sound reflection etc). Suppose I want to have a nice reflection in a pool of water for example, you can already get a very good looking effect by raytracing a low-resolution environment map from the water pools' perspective, which wouldn't need any tracing of secondary rays etc, and use it in your water shader.
There's uncountable ways something like this could be useful (not all of which can be accelerated very well though, I guess)
For doing the more traditional raytracing at any resolution you will always need an acceleration structure like a BVH though, but building/updating that is part of these kinds of hardware extensions.
The more the API situation evolves, I can see Google, Microsoft and Sony going all in all Vulkan for performance graphics. Only if MS is willing to part ways with D3D12
For cross platform I could see WebGPU + WebAssembly being the choice, where it abstracts both Metal and Vulkan.
Vulkan is a PC thing, without any impact on consoles, even on the Swift, NVN is what gets mostly used unless it is a PC port.
Also the subject of this thread, Ray Tracing, was developed by Microsoft and NVidia together and took two years to reach Vulkan.
How long will Vulkan now take to adopt shader meshes?
That leaves Vulkan for Linux games. The 1% of the market that no savvy business person would touch with a 10 feet pole.
In Linux land, you'll meet people who feel suspicious of you in the first place for charging for your game, and people who have compiled their GPU driver from scratch, yet blame your game for being incompatible with their wrong compiler flags.
So for almost every game studio, there is no viable market where you could sell a Vulkan game.
This pretty much only leaves out macOS and iOS, for which a Vulkan compatibility layer exists anyways.
That would limit Vulkan as a good cross-platform choice to the few developers that do not use Unity or Unreal Engine under the hood.
Nintendo uses Vulkan on the Switch, but they also have their own Metal like API, NVN.
Vulkan is mostly used by PC ports into the Switch.
Exclusive Switch titles do so via NVN, Unreal or Unity bindings to NVN, not Vulkan.
Android has optional support for Vulkan, since only flagship devices ever bothered to actually provide it, with Android 10 Google has changed it into compulsory API. So only Android 10 or later devices are guaranteed to actually have proper Vulkan drivers available.
Windows supports Vulkan on the classic OpenGL ICD driver model, not available in Win32 and UWP sandboxes, which only allow for DirectX based code.
This is the actual reality, and not that typical Vulkan sales pitch.
I like this. I will use the word "sales pitch" next time other companies does it, which to me is flat out lying. ( Cough; AV1 )
I still dont get why the hype on Vulkan, if you want Android support you will still need to use OpenGL for the majority of devices, given their slow replacement cycle and I doubt it will be mainstream anytime soon.
On top of that, Vulkan is much hard to actually use in practice. It's much more low-level than OpenGL and DirectX.
this might be why WebGL2 never really shipped on Safari. They added a flag but there is no actual implementation.
If this actually the reason, I don't know, corrections welcomed.