Skip to content

developer.md

Latest commit

 

History

History
720 lines (615 loc) · 45.7 KB

developer.md

File metadata and controls

720 lines (615 loc) · 45.7 KB
Raw

Developer Guide -- Vulkan Ray Tracing glTF Renderer

For contributors and agents. High-level architecture, source map, and the material system — concepts plus pointers into src/ and shaders/, not a restatement of the code. For runtime usage see the User Guide; for the scene data-flow deep dive see Rendering Architecture.

This project is developer-oriented: the RTX path tracer is the primary high-quality reference for ray tracing glTF PBR materials, while the rasterizer is a practical fallback for fast interaction and iteration. Both renderers share Vulkan resources and Slang shaders.

Quick Navigation

For the internal data-flow walkthrough (Model -> RenderNodes -> GPU/TLAS), including how ray tracing acceleration structures are built, see Rendering Architecture.

For how glTF 2.1 external assets (complex scenes) are merged, flagged read-only, and re-externalized/flattened on save, see External Assets. For glTF 2.1 support as a whole (status, PROVISIONAL markers, the tinygltf fork, test fixtures), see glTF 2.1.

Architecture

Application Framework

nvapp::Application provides the Vulkan application lifecycle. You attach nvapp::IAppElement instances, and each is called for different states:

Callback Purpose
onAttach() Create Vulkan resources, compile shaders, set up descriptor sets
onDetach() Clean up all GPU resources
onRender(cmd) Record rendering commands (dispatch to path tracer or rasterizer, then tone map)
onResize() Recreate G-Buffer and notify renderers
onUIRender() Render ImGui UI and handle input
onUIMenu() Create menu bar (File, View, Tools, Debug)
onFileDrop() Load dropped .gltf/.glb/.obj (or a .scene.json descriptor) as scenes, .hdr as environment maps

In main() we attach:

  • GltfRenderer — the main element (scene loading, rendering, UI)
  • ElementProfiler — GPU execution timing
  • ElementLogger — log output in a UI window
  • ElementGpuMonitor — NVML-based GPU status monitoring
  • ElementDbgPrintf — shader printf debug output (only when built with USE_DBG_PRINTF)

Path-traced depth

On the first frame after a reset, the path tracer writes NDC depth (outDepth) so picking and the overlays drawn after it (grid, gizmo) can depth-test against the path-traced image. It writes the G-buffer depth image directly as a storage image when the depth format supports it. nvvk::RenderTarget only grants STORAGE usage in that case, and Vulkan does not require it for any depth format; D32_SFLOAT lacks it on AMD. There PathTracer binds an R32_SFLOAT staging image instead and copies it into the depth image through a buffer (PathTracer::cmdCopyStagedDepth), since core Vulkan has no direct copy between color and depth images.

Scene Graph

The glTF scene is loaded via TinyGLTF and converted to a GPU-friendly representation shared by both renderers.

---
config:
  layout: elk
  theme: neutral
---
flowchart TB
 subgraph RenderNodes["RenderNodes"]
        rn0["rn0"]
        rn1["rn1"]
        rn2["rn2"]
        rn3["rn3"]
        rn4["rn4"]
  end
 subgraph s1["RenderPrimitives"]
        n1["rp0"]
        n2["rp1"]
  end
 subgraph s2["RenderMaterials"]
        n3["m1"]
        n4["m2"]
        n5["m3"]
  end
    sceneRoot["sceneRoot"] --> node0["node0"]
    node0 --> node1["node1"] & node2["node2"] & node3["node3"]
    node3 --> mesh1["mesh1"]
    mesh0["mesh0"] --> prim0["prim0"] & prim1["prim1"]
    mesh1 --> prim2["prim1'"]
    node1 --> mesh0
    node2 --> mesh0
    prim0 --> rn0 & rn2 & n1 & n3
    prim1 --> rn1 & rn3 & n2 & n4
    prim2 --> rn4 & n2 & n5

    style node0 fill:#BBDEFB
    style node1 fill:#BBDEFB
    style node2 fill:#BBDEFB
    style node3 fill:#BBDEFB
    style prim0 fill:#C8E6C9
    style prim1 fill:#C8E6C9
    style prim2 fill:#C8E6C9
    style RenderNodes fill:#BBDEFB
    style s1 fill:#C8E6C9
    style s2 fill:#E1BEE7
Loading

RenderNodes are the flattened instances with world matrices and material references. RenderPrimitives are the unique geometry (vertex/index buffers). Shared primitives create one BLAS for multiple render nodes.

RN0 == node0 -> node1 -> prim0 RN1 == node0 -> node1 -> prim1 RN2 == node0 -> node2 -> prim0 RN3 == node0 -> node2 -> prim1 RN4 == node0 -> node3 -> prim1'

Source Code Structure

Primary files grouped by subsystem (not an exhaustive listing — treat src/ and shaders/ as the source of truth for the complete set):

src/
├── main.cpp                    # Entry point, Vulkan context, CLI parsing
├── renderer.cpp/hpp            # Main renderer orchestrator (GltfRenderer)
├── renderer_base.hpp           # BaseRenderer: abstract base for PathTracer / Rasterizer
├── renderer_pathtracer.cpp/hpp # Monte Carlo path tracer (Vulkan ray tracing + ray query)
├── sample_schedule.hpp         # Samples per frame: ending on stops (cap, --outputSamples), the Auto SPP controller
├── renderer_rasterizer.cpp/hpp # Forward PBR rasterizer
├── renderer_silhouette.cpp/hpp # Selection highlight (compute shader)
├── env_image_loader.cpp/hpp    # Decodes .hdr / .exr lat-longs to float RGBA for HdrIbl
├── env_baker.cpp/hpp           # Bakes an analytic sky into the lat-long image + importance buffer
│                               #   that HdrIbl / HdrEnvDome consume (preview vs. commit)
├── gltf_environment_sky.cpp/hpp# OMI_environment_sky parse/write/strip, raw-value passthrough
├── mcp_timing.cpp/hpp          # Optional MCP endpoint (--mcp): recompile shaders, time a
│                               #   GPU pass over a window (docs/mcp.md)
├── hover_picker.cpp/hpp        # Async G-buffer readback for KHR_interactivity hover detection
├── camera_controls.cpp/hpp     # Viewport bindings table (drags, wheel, clicks, keys -> action), the manipulator
│                               #   that dispatches from it (Shift pivots on the ray-picked point), and
│                               #   the per-frame input handling; Help's Camera tab reads the same table
├── settings_registry.hpp       # One declaration per setting -> CLI + benchmark sequences + ImGui.ini;
│                               #   also captures each declaration-time default (Reset All to Default)
├── resources.hpp               # Shared Vulkan resources and settings (`nvvk::FrameUploader` staging)
│
├── gltf_scene.cpp/hpp          # Core scene loading and management
├── gltf_scene_vk.cpp/hpp       # GPU buffer/texture upload (SceneVk)
├── gltf_scene_omm.cpp/hpp      # Opacity micromaps (SceneOmm), EXT_mesh_opacity_micromap → VK_KHR_opacity_micromap
├── gltf_scene_rtx.cpp/hpp      # BLAS/TLAS acceleration structures (SceneRtx)
├── gltf_scene_gpu.cpp/hpp      # SceneGpu — coordinates SceneVk / AnimationVk / SceneRtx / TransformComputeVk
├── gltf_scene_editor.cpp/hpp   # Scene editing (add/delete/duplicate nodes)
├── gltf_scene_animation.cpp/hpp# AnimationSystem: playback/interpolation (CPU), via Scene::animation()
├── gltf_scene_animation_vk.cpp/hpp # GPU compute skinning / morph target animation
├── gltf_scene_transform_vk.cpp/hpp # GPU compute world-matrix propagation
├── gltf_scene_merger.cpp/hpp   # Merge two glTF scenes; append a file's resources (external assets)
├── gltf_scene_external_assets.cpp # glTF 2.1 external assets: file cache, placement, instance sets (ExternalAssetPlacer)
├── gltf_scene_loading.hpp      # Shared file loading helpers (aliases, image rebase, meshopt)
├── gltf_scene_validator.cpp/hpp# Scene validation
├── gltf_version.cpp/hpp        # glTF asset versions: what loads, what a save declares, 2.1 feature detection
├── gltf_compact_*.cpp/hpp      # Orphan resource removal / buffer compaction
├── gltf_create_tangent.cpp/hpp # Tangent space generation (UV gradient + MikkTSpace)
├── gltf_material_cache.cpp/hpp # CPU-side material cache
├── gltf_image_loader.cpp/hpp   # Image decoding (DDS, KTX, STB, WebP)
├── gltf_animation_pointer.*    # KHR_animation_pointer support
├── ies_profile.cpp/hpp         # EXT_lights_ies: IESNA LM-63 photometric profile parser
│
├── ui_inspector.cpp/hpp        # Property inspector (per selection type: node/primitive/material/
│                               #   mesh/camera/light/texture/image/sampler/animation)
├── ui_scene_browser.cpp/hpp    # Scene Browser window: Scene Graph tree + the Elements tab host
├── ui_scene_browser_elements.cpp # Elements tab: builds the ElementTypeDesc registry and the one
│                               #   generic list renderer (icon tabs, columns, toolbar, sort, filter)
├── ui_element_registry.hpp     # ElementTypeDesc / ElementColumn / ElementAddVariant (data-driven list)
├── ui_gltf_labels.cpp/hpp      # Shared image-name + sampler wrap/filter labels (list + inspector)
├── ui_renderer.cpp             # Viewport UI, menus and mouse interaction
├── ui_environment.cpp/hpp      # UiEnvironment: the Environment panel. Environment type, HDR file,
│                               #   the authored skies, the atmosphere folds, Sun & Time of Day.
│                               #   Takes what it needs through setters, like UiInspector
├── sky_sun.cpp/hpp             # SkySun: who the sun is (renderer state, or a scene light marked
│                               #   as the sky's) and every path that aims it
├── sun_position.cpp/hpp        # NOAA solar position: (place, UTC instant) -> bearing + altitude.
│                               #   Pure, self-contained, nothing else links against it
├── ui_dock_layout.cpp/hpp      # Default docking arrangement, shared by start-up (main.cpp's
│                               #   dockSetup) and Windows > Reset UI Layout
├── ui_presentation_mode.cpp/hpp # Presentation mode (F11): hides panels + menu bar, borderless
│                               #   full screen on the window's monitor; saves/restores both
├── ui_commands.cpp/hpp         # Command registry: every menu/toolbar/keyboard command and its
│                               #   key binding, and the shortcut routing (see Add a command)
├── ui_help.cpp/hpp             # Help menu: Keyboard & Mouse reference (live camera table,
│                               #   searchable shortcuts generated from the registry) and About
├── ui_xmp.cpp/hpp              # KHR_xmp_json_ld metadata display
├── ui_animation.cpp/hpp        # Animation playback state and viewport animation widget
├── ui_mouse_state.hpp          # Mouse state tracking for viewport
├── ui_busy_window.hpp          # Loading/busy indicator overlay
│
├── dlss.cpp/hpp                # DLSS adapter (Ray Reconstruction + Super Resolution)
├── dlss_wrapper.cpp/hpp        # NGX / DLSS SDK wrapper (project-independent)
├── optix_denoiser.cpp/hpp      # OptiX AI Denoiser integration
├── oidn_denoiser.cpp/hpp       # Open Image Denoise on Vulkan (vk_oidn) integration
├── denoiser_schedule.hpp       # Auto-denoise interval shared by OptiX and OIDN, DLSS hand-off weight
├── denoiser_handoff.cpp/hpp    # DLSS hand-off: blend from DLSS-RR's image to OptiX / OIDN's
├── vk_cuda.cpp/hpp             # Vulkan-CUDA interop (shared memory/semaphores)
│
├── gizmo_transform_vk.cpp/hpp  # 3D translate/rotate/scale gizmo
├── gizmo_grid_vk.cpp/hpp       # Infinite procedural grid
├── gizmo_visuals_vk.cpp/hpp    # Helper overlay manager
│
├── undo_redo.cpp/hpp           # Full undo/redo system for scene editing
├── tinygltf_utils.cpp/hpp      # TinyGLTF material + node extension extraction (visibility/selectability/hoverability)
├── tinygltf_converter.cpp/hpp  # OBJ to glTF converter
├── tiny_stb_implementation.cpp # STB library implementation (single compilation unit)
├── pipeline_cache_util.cpp/hpp # Vulkan pipeline cache persistence
├── scene_selection.cpp/hpp     # Selection state and events
├── scene_descriptor.cpp/hpp    # Scene descriptor set layout / bindings
├── scene_feature_detection.cpp/hpp # Detect which extensions a scene uses
├── scene_shader_macros.cpp/hpp # Emit -DGLTF_USE_* macros from detected features
├── timeline_pipeline.cpp/hpp   # Timeline-semaphore submission helper
├── benchmarking.cpp/hpp        # Headless timing / scripted GPU benchmarks
├── gltf_camera_utils.hpp       # Camera utility functions
├── gpu_memory_tracker.hpp      # GPU memory category tracking
├── scoped_banner.hpp           # Scoped UI banner helper
└── utils.hpp                   # Miscellaneous utilities

shaders/
├── gltf_pathtrace.slang        # Path tracer (compute + ray generation)
├── gltf_raster.slang           # PBR rasterizer (vertex + fragment)
├── silhouette.comp.slang       # Selection edge detection
├── skinning.comp.slang         # GPU skinning
├── morph.comp.slang            # GPU morph-target blending
├── world_matrix_propagate.comp.slang   # Per-level world-matrix propagation
├── snapshot_prev_transforms.comp.slang # Snapshot previous transforms (motion vectors)
├── update_render_instances.comp.slang  # Refresh render-instance data on GPU
├── animation_io.h.slang        # Skinning/morph shader I/O
├── world_matrix_io.h.slang     # World-matrix compute shader I/O
├── shaderio.h                  # Host/device shared structures
├── pathtrace_functions.h.slang # Path-tracer hit/shading helpers (getShadowTransmission, etc.)
├── common.h.slang              # Shared shader utilities
├── get_hit.h.slang             # Hit state interpolation
├── raytracer_interface.h.slang # RayQuery vs traditional RT abstraction
├── dlss_util.h                 # DLSS motion vector / guide buffer utilities
├── gizmo_grid.slang            # Infinite grid rendering
├── gizmo_grid_shaderio.h.slang # Grid shader I/O
├── gizmo_visuals.slang         # Transform gizmo overlays
├── gizmo_visuals_shaderio.h.slang # Gizmo shader I/O
├── optix_image_to_buffer.slang # OptiX denoiser buffer conversion
├── denoiser_handoff.slang      # DLSS hand-off blend of two images
├── gltf_light_ies.h.slang      # EXT_lights_ies: photometric-profile falloff lookup
├── sky_background.h.slang      # Background (sun-bearing) field for authored skies; shared by both paths
├── sky_background.slang        # Rasterizer background pass that draws it
├── sky_omi_shaderio.h.slang    # SkyType + the per-type OMI_environment_sky parameter blocks
├── sky_gradient.h.slang        # OMI `gradient` evaluator (bands + sun), shared by bake and background
├── env_bake.slang              # Bakes any sky into the lat-long image + importance buffer; one
│                               #   switch, with the physical case reading SkyBruneton's LUTs
│
│   # Local material fork (see "Material System" below)
├── gltf_material_config.h      # MAT_EXT_* compile-time feature flags
├── gltf_eval_config.h          # GLTF_USE_* runtime behavior gates
├── gltf_scene_io.h.slang       # GltfShadeMaterial / GltfLight / GltfScene shaderio structs
├── gltf_vertex_access.h.slang  # Vertex buffer accessors
└── gltf_material_eval.h.slang  # evaluateMaterial(): GltfShadeMaterial -> PbrMaterial

Material System

The glTF material struct (shaderio::GltfShadeMaterial) and its evaluateMaterial() function are forked locally into shaders/gltf_*.h.slang rather than using the upstream nvpro_core2/nvshaders/ versions. The fork exists for two reasons:

  1. Forkability: the struct layout can be tailored to this project without affecting other samples that share nvpro_core2.
  2. Extension gating: every KHR_materials_* extension (plus KHR_texture_transform) is controlled by a compile-time MAT_EXT_* flag in shaders/gltf_material_config.h. When a flag is 0, that extension's fields drop out of the relevant struct (GltfShadeMaterial for KHR_materials_*, GltfTextureInfo for KHR_texture_transform) and its code paths drop out of evaluateMaterial() / getTexture() / getShadowTransmission(). Defaults are all 1, reproducing the upstream behavior byte-for-byte.
  3. Behavior gating (runtime-safe): GLTF_USE_* in shaders/gltf_eval_config.h gates material eval, path-tracer state (PathTracerState, volume segment, shadow transmission), and raster extension paths without changing buffer layout. Optimal mode passes each gate as -DGLTF_USE_X=0 or 1 from SceneFeatureSet (never -DMAT_EXT_X=0 at runtime). MAT_EXT_* remains layout authority on host and device only.

Key design points

  • gltf_scene_io.h.slang and gltf_vertex_access.h.slang intentionally reuse the upstream include guards (DH_SCN_DESC_H, VERTEX_ACCESSORS_H) so that transitive includes from nvpro_core2 headers (e.g. nvshaders/light_contrib.h.slang) become no-ops. The local version must be included first in every translation unit.
  • gltf_material_eval.h.slang uses its own guard (GLTF_MATERIAL_EVAL_H) because no upstream code transitively includes it.
  • PbrMaterial and the BSDF interface (nvshaders/bsdf_functions.h.slang) stay in nvpro_core2. The local evaluateMaterial() initializes PbrMaterial with defaultPbrMaterial() first, then overrides per extension — gating an extension off safely leaves the corresponding BSDF fields at their neutral defaults.
  • Struct members carry their defaults inline, so GltfShadeMaterial m = {}; works; no parallel default*() helpers to keep in sync (see shaders/gltf_scene_io.h.slang).
  • Host-side layout is guarded by static_asserts in src/gltf_material_cache.cpp (8-byte alignment/stride plus anchor offsets), so flipping any MAT_EXT_* flag that breaks the layout fails the build immediately.
  • Retroreflection is a worked example of the pattern: MAT_EXT_RETROREFLECTION gates KHR_materials_retroreflection on both host and device, with the evaluation living in gltf_material_eval.h.slang. See the KHR_materials_retroreflection specification.

Texture channels: KTX2 swizzle and two-channel sources

Container formats can store fewer channels than the shader expects, in two different ways, and the renderer resolves both on the host so the shader stays simple.

  • Authored swizzle. A KTX2 file may carry a KTXswizzle key (e.g. a BC5 metallic-roughness map that declares 1rg1). nv_ktx parses it into KTXImage::swizzle; gltf_image_loader.cpp converts it to a VkComponentMapping, and SceneVk::createImage() applies it to the VkImageView — so the swizzle costs nothing at sample time and both render paths see the remapped channels. The same field is used to expand single-channel stb_image results to grayscale.
  • Missing channels. A two-channel format (BC5, R8G8, EAC_R11G11) has no third channel to swizzle from; Vulkan samples blue as a synthetic 0. That is fine for data that only occupies two channels, but it breaks tangent-space normal maps, whose Z is implied rather than stored — decoding the synthetic 0 to -1 flips the normal into the surface and the surface renders black. SceneVk::buildTextureFormatFlags() infers the component count from the decoded VkFormat and marks the exactly two-channel ones in GltfTextureInfo::flags; decodeTangentSpaceNormal() in gltf_material_eval.h.slang rebuilds Z from the unit-length assumption. Three-channel maps keep their authored Z, and single-channel formats are excluded — their green is synthetic too, so there is no second axis to reconstruct from.

Because the flag depends on the decoded image format, texture creation runs before uploadMaterials() in SceneVk::create() and recreatePreservingTextures() — the material cache bakes the per-texture sampler slot and format flags (TextureSlotTable) into GltfTextureInfo.

Worked example: KHR_materials_scatter (two modes, one material block)

KHR_materials_scatter is the most involved extension in the fork because it changes either the surface BSDF or the volume transport, depending on the material. Both branches live in the scatter block of gltf_material_eval.h.slang; the mode is selected by PbrMaterial::thickness (i.e. by KHR_materials_volume), matching the extension spec.

  • Thin-walled (no volume, or thickness 0). The spec turns the transmissive part of the dielectric base into a blend of the specular BTDF and a Lambertian BSDF that splits its energy between the two hemispheres by the scatter anisotropy. There is no dedicated scatter lobe in nvpro_core2's BSDF, so the block re-expresses that blend with the lobes that already exist. It moves transmission * scatterStrength out of the specular BTDF into the diffuse bucket, then splits that bucket with diffuseTransmissionFactor. Note what the factor is: the forward share of the whole diffuse bucket, not the anisotropy's forward fraction — the bucket also holds the untouched diffuse base, so the fraction has to be normalised by the bucket's own weight. Conflating the two is what makes the mapping wrong for transmissionFactor < 1, and it is only by coincidence correct at transmissionFactor = 1. Done that way the lobe weights match the spec exactly for any transmission, strength and anisotropy. The spec also gives the residual specular BTDF and the diffuse reflection lobe different colors, and PbrMaterial::baseColor alone can only carry one — hence PbrMaterial::transmissionTint in nvpro_core2, a multiplier on the BTDF lobe that is float3(1) for every other material.
  • Volumetric (thickness > 0). The surface BSDF is untouched; the block only derives PbrMaterial::scatterCoefficient (σs) by splitting the KHR_materials_volume extinction with the Kulla-Conty single-scatter albedo. The transport that consumes it is handleVolumeScatter() in pathtrace_functions.h.slang.

Some things about the transport are worth knowing before touching it:

  • glTF media are homogeneous, so the free-flight distance is sampled analytically per spectral channel and combined with the balance heuristic — not with delta tracking against a majorant. A single majorant makes the low-extinction channels collide (and therefore random-walk) far too often; because those extra collisions are lossless, their effective per-collision albedo drifts towards 1 and the medium's perceived color shifts away from multiscatterColor.
  • VolumeMedium stores absorption, not extinction, alongside the scattering coefficient. A medium authored near white has σa as a tiny difference of two large numbers — recovering it by subtraction under limited precision quantizes the albedo straight to 1.
  • In-volume Russian roulette has its own probability cap (RR_PCONT_CAP_VOLUME). A dense, high-albedo medium keeps the throughput near 1, so the surface cap would cull fully energetic paths at every scatter event and the medium would render far too dark. The cap must stay below 1 all the same: without roulette such a path never terminates.
  • Surfaces with a diffuse transmission lobe are lit from both hemispheres, so sampleLights() takes a flag that disables the punctual-light hemisphere rejection in singleLightContribution(). Without it, a light behind a backlit leaf or sheet contributes nothing. For the same reason SceneRtx disables back-face culling on them (getInstanceFlag()): a diffusely transmitted ray reaches the far wall of a closed object from behind.
  • Straight-through shadow transmission (getShadowTransmission()) only passes the specular part of the transmission, so in thin-walled mode it is scaled by 1 - scatterStrength. It samples the transmission and scatter-strength textures at the hit, so a shadow ray agrees with evaluateMaterial() texel by texel.
  • The thickness texture is a rasterizer-only stand-in for the path length through the medium: evaluateMaterial() applies it under RASTER_PIPELINE only. In the path tracer the ray measures that length, and thicknessFactor alone selects thin-walled vs. volume. It is also what makes a surface a medium boundary: the path and its shadow rays count themselves inside only across surfaces with a thickness.
  • Light reaching the camera through a medium is attenuated wherever it comes from. The medium in front of a hit (processVolumeSegment()) is resolved before the surface's emission is added, and a shadow ray knows the medium it starts in (the path's, from a surface or a scatter point) and attenuates every segment it travels inside one, including the last, up to a light inside the medium. tests/render/volume_transmittance.py checks both against analytic transmittance.
  • A path knows it is in a medium only by having crossed into one, so a camera path starts in the medium the camera is in: once per accumulation of a scene with a KHR_materials_volume material, PathTracer::probeCameraMedium() runs the path tracer's own pipeline for a single thread (probeCameraMedium() in gltf_pathtrace.slang), which follows the center camera ray to the nearest medium boundary and, if it meets it from behind, records that medium for pathTrace() to start in. Without it, a camera inside a fog sees no fog in front of it and fog behind the far wall.

The rasterizer cannot trace inside a medium, so evaluateMaterial() compiles only the thin-walled branch for RASTER_PIPELINE — the approximation the scatter spec explicitly permits for renderers without volumetric transport.

Adding a new KHR_materials_* extension

  1. Define a new MAT_EXT_<NAME> flag in gltf_material_config.h, default 1.
  2. In gltf_scene_io.h.slang, add the new struct fields (+ any texture slots) inside #if MAT_EXT_<NAME> with inline default values.
  3. Add GLTF_USE_<NAME> in gltf_eval_config.h (default MAT_EXT_<NAME>) and the matching #error guard.
  4. In gltf_material_eval.h.slang, add the evaluation block inside #if GLTF_USE_<NAME>, overriding the relevant PbrMaterial fields. Respect the existing block order (volume block runs before IOR, etc.).
  5. In src/gltf_material_cache.cpp, add the matching CPU-side population block under #if MAT_EXT_<NAME> — layout only; never toggle MAT_EXT_* at runtime.
  6. In src/scene_feature_detection.cpp and src/scene_shader_macros.cpp, detect scene usage and emit -DGLTF_USE_<NAME>=0|1 in optimal mode.
  7. If the path tracer or raster reads the new field outside evaluateMaterial() (e.g. getShadowTransmission() in pathtrace_functions.h.slang), gate with #if GLTF_USE_<NAME>.
  8. Update documentation in README.md (extension support list) and user-guide.md.

Adding a non-PBR material extension (guide-buffer pattern)

Some material extensions don't influence the BSDF at all — they feed denoiser / post-process inputs instead. EXT_DLSS_NR is the reference implementation: it writes a 4-channel per-material mask into a dedicated DLSS-NR guide-buffer slot rather than touching evaluateMaterial(). The checklist differs from the PBR pattern above:

  1. MAT_EXT_* flag + struct field in gltf_material_config.h / gltf_scene_io.h.slang (same as PBR, but no gltf_eval_config.h / gltf_material_eval.h.slang entry needed).
  2. Add a GBuffer slot to OutputImage enum (shaders/shaderio.h) and the matching VkFormat to kRrInnerFormats[] (src/dlss.cpp).
  3. Write the slot in processPixel (shaders/gltf_pathtrace.slang) inside the first-hit block, guarded by USE_DLSS_SHADER && MAT_EXT_<NAME>. Propagate through GuideScratch → GuideOutput → packSampleResult (shaders/pathtrace_functions.h.slang) if the value needs to survive path tracing.
  4. Register the new slot in kRrSlots[] (src/renderer_pathtracer.cpp) so it is wired into the outImages bindless array.
  5. Wire the new NGX input in Dlss::evaluateNr() or the appropriate evaluate function (src/dlss.cpp), reading directly from m_innerGBuffer.
  6. tinygltf helpers (src/tinygltf_utils.{hpp,cpp}), material cache loading (src/gltf_material_cache.cpp), extension registration (src/gltf_scene.cpp), and inspector UI (src/ui_inspector.{hpp,cpp}) follow the same pattern as PBR extensions — see EXT_DLSS_NR for the complete example.
  7. Update docs/denoising.md (for guide-buffer / denoiser integration) and user-guide.md.

Lighting

EXT_lights_ies provides photometrically accurate angular light distribution using IESNA LM-63 profile files. Per spec it defines a self-sufficient point light and does not compose with KHR_lights_punctual. If a node carries both extensions, EXT_lights_ies wins and the KHR attachment on that specific node is dropped at load with a warning (see Scene::handleLightTraversal in src/gltf_scene.cpp); the shared KHR light entity in model.lights[] is preserved, so other nodes referencing it are unaffected.

IES profiles — ring and scatter distributions on floor and wall

src/ies_profile.hpp parses the .ies file into a normalized, azimuthally-averaged 1D candela curve (see its "Simplification" note on why genuinely asymmetric fixtures lose their azimuthal variation); GltfScene.iesProfiles carries the flattened per-profile tables to the GPU, and GltfLight.iesProfile indexes into it (see shaders/gltf_scene_io.h.slang). Both render paths evaluate the profile identically via evalIesFactor() in shaders/gltf_light_ies.h.slang.

The Inspector exposes multiplier and color as editable property-editor rows on any node carrying EXT_lights_ies (UiInspector::renderNodeProperties -> renderIesEditor). Edits are undoable via EditNodeIesCommand. The profile index itself is read-only (it refers to the loaded .ies file array). Nodes that additionally carry KHR_lights_punctual show an inline warning in the IES section explaining that the KHR attachment is ignored on that node. See user-guide.md § IES Photometric Profiles for profile comparison renders and the inspector UI.

Environment Lighting

Settings::envSystem (shaderio::EnvSystem) selects the environment source. The shaders do not branch on it: every mode but eNone reaches them as one lat-long image plus its alias table (Resources::hdrIbl) and, for the rasterizer, the prefiltered cubes derived from it (Resources::hdrDome). That is the whole point of the design — sampling, MIS and raster IBL stay free of per-sky-type code, and adding a sky type adds no shader branches.

An HDR file arrives there by being loaded. The analytic types — the physical sky and the OMI_environment_sky types — arrive there by being baked: src/env_baker.* dispatches shaders/env_bake.slang — one switch over every sky type — into a lat-long image and an importance buffer, then hands the pair to nvvk::HdrIbl::updateFromGpuImage and nvshaders::HdrEnvDome::updateEnvironment. The bake's bindings and push constant are ours (shaders/env_bake_io.h.slang); what nvpro_core2 fixes is the lat-long mapping and texel weighting (latlongToDir, texelSolidAngle in nvshaders/functions.h.slang), next to the functions that read the image back. What a sky is stays in this repository, next to the glTF loader that reads it.

The bake size is fixed (kEnvBakeSize in src/env_baker.hpp), and the alias table is not built at it. The bake sums its importance per cell of a coarser sampling grid (kEnvSamplingGrid, same header) — a shared-memory reduction inside each workgroup, so it costs next to nothing — and updateFromGpuImage builds the table over the cells, so its cost is the grid's, not the image's. A sample picks a cell and then a direction uniformly in solid angle within it, and the PDF written into the image's alpha is the cell's, so sampling and MIS agree. That suits a baked sky, whose lighting field is smooth because its sun is a light of its own; an HDR file keeps one entry per texel, since a photographed sun is a few texels wide. Shaders therefore sample with the grid (SceneFrameInfo::envSamplingGridWidth/Height), never the image size. EnvBaker splits that into a cheap preview() (colour only, while something is moving) and a full commit(). Which one runs is decided by GltfRenderer::requestEnvironmentPreview(): every edit source reports through it, and the commit is paid on the first frame that asks for no preview. That settle test, rather than an ImGui "is a control held" question, is what keeps a sun driven by the gizmo or by an animation from committing once per frame. GltfRenderer::updateHdrImages() remains the single place that writes the texturesCube[] / texturesHdr[HDR_IMAGE_INDEX] descriptors.

The image, its alias table and the cubes are all stored in the environment's own frame. One orientation, SceneFrameInfo::envRotation — a unit quaternion, OMI_environment_sky's rotation verbatim — takes that frame to world. Every lookup brings a world direction into it with quatRotateInverse (nvshaders/functions.h.slang), and nvpro_core2's raster IBL (evalKhronosIBL) and HdrEnvDome::draw take the same quaternion. Rotating the environment therefore never re-bakes; only the physical sky does, when the sun moves relative to it.

Two fields: lighting and background

The environment is two fields, and which one a ray reads is the part that has to be right:

  • Lighting field — the baked lat-long image, deliberately sun-free. The sun is a directional KHR_lights_punctual that next-event estimation samples directly; baking it into the image too would light every rough surface with roughly two suns.
  • Background field — evaluated analytically per ray, sun included. Per-ray evaluation is also what keeps a half-degree sun disk sharp, which no practical bake resolution could. The authored sky types evaluate it in shaders/sky_background.h.slang. The physical sky's background is evaluated per ray in the path tracer too (evalPhysicalSkyBackground in shaders/gltf_pathtrace.slang), through the same function the bake uses (shaders/sky_bruneton_radiance.h.slang), plus the half-degree disk (evalSunDiskBackground in the same header, fed by a radiance the bake publishes). What the camera sees is then independent of the bake resolution — the horizon, where sky meets ground in a hard edge, is what a lat-long would otherwise smear. The rasterizer still draws the baked image for this sky and adds the disk, so its horizon keeps the bake's resolution.

A camera ray, and an escaped ray whose last bounce was delta/specular, read the background field; any other escaped ray reads the lighting field plus the sun disk (evalSkySunDisk), MIS-weighted against the sun's NEE sample. The disk is needed there because a glTF roughness of 0 is clamped to a tiny but finite lobe, not a delta: NEE's cone samples almost never land in it, so without the disk smooth glass or a mirror shows no sun. The header states the rule at its definition because getting it wrong is silent and off by roughly 2x. Both render paths include that same header — the path tracer from gltf_pathtrace.slang, the rasterizer from its background compute pass shaders/sky_background.slang — so the two can never drift.

Panorama: where the image URI lives

OMI's panorama.equirectangular is an index into the glTF's textures[], not a path. An .hdr or .exr is not a standard glTF image mime type, and putting one into images[]/buffers[] to satisfy that would bloat every scene that has a sky — so this renderer stores a URI in a sibling NV_environment_sky_panorama block on the same sky entry:

"skies": [{
  "type": "panorama",
  "extensions": { "NV_environment_sky_panorama": { "uri": "../env/studio.exr" } }
}]

The interop gap is deliberate and worth stating: a reader that understands only OMI sees type: "panorama" with no source and falls back to its own default. That is a visible gap rather than a silent wrong answer, which is the better failure. An authored equirectangular index is never invented but always preserved, since index 0 is a valid texture — inventing one would load the wrong image rather than fall back.

The URI resolves against the glTF that named it, like every other external asset (external_assets.md), not against the resources directories — two scenes naming the same file must not load each other's environment. On save it is written relative to the file being written when the image lives under it, and absolute when it does not, so a scene and its environment move together without a chain of .. that breaks at the first move.

Decoding is src/env_image_loader.{cpp,hpp}, which is also the single place that decides which formats can be an environment. It dispatches on content rather than file extension, and covers both .hdr and .exr — see that header for why EXR cannot live inside nvvk.

Where the sky lives in the scene

OMI_environment_sky is scene data, not a viewer preference. src/gltf_environment_sky.* parses the extension into a SkyDescriptor and writes it back on save, keeping the raw tinygltf::Value so sky types and keys the renderer does not model survive a round trip untouched. GltfRenderer applies a parsed sky at the top of the frame after the settings-restore hooks have run, so a scene that carries a sky wins over the persisted .ini.

Agentic Workflow Bridge

The optional external generation bridge lives in src/agentic_bridge.*. It defines a filesystem-based manifest and request/response envelope for tools that can generate HDR environments, texture references, or image-to-image enhancements.

Initialize the bridge with:

vk_gltf_renderer --agenticBridgeInit --agenticBridgeRoot path/to/agentic_bridge

This creates manifest.json, requests/, responses/, and assets/ without requiring ComfyUI or any other generator to be installed. The same bridge can be initialized and driven from the in-app Agentic window (F7, in the Windows menu). The C++ is compiled only when the USE_AGENTIC CMake option is ON (the default). See Agentic Workflow Foundation for the staged plan and protocol shape.

Common Development Workflows

Add a command-line parameter

  1. Register it once: m_settings.add(...) (SettingsRegistry) for a renderer setting, or parameterRegistry.add(...) in main.cpp for an app-level flag.
  2. Wire it to renderer or app settings.
  3. Nothing to list by hand: it appears in --help. Describe it in the user guide only where users need to know how to use it.

Add a command or a keyboard shortcut

Commands live in one place: GltfRenderer::registerCommands() (ui_renderer.cpp) adds each to the ui::CommandRegistry (ui_commands.hpp), with its label, icon, menu group, what it does, and optionally when it is enabled or checked. Its key binding sits beside it in the same declaration, held apart from the command so bindings can later become user-settable. Everything else is a view of the registry: a menu draws it with menuItem(id) (icon, label and shortcut text come along), the toolbar draws window and view toggles from it, and Help > Keyboard & Mouse lists every bound command, grouped by menu. A new shortcut therefore needs no change outside registerCommands().

Keys go through Dear ImGui's shortcut routing (ImGui::Shortcut()), polled once per frame per binding scope: global bindings by the menu bar, viewport bindings from inside the viewport window (live while the viewport is hovered — no click needed — or focused), and F11 from updatePresentationMode(), since the menu bar is hidden in presentation mode. That function also polls the presentation-only bindings (Esc leaves the mode) while the mode is on, so they never claim a key the rest of the time. Global bindings take the lowest routing priority, so a widget using the keyboard keeps it: while a text field is edited, letters, Delete, Home and Esc go to the field, and its own Ctrl+Z / Ctrl+Y win over Undo / Redo. Two kinds of binding are listed but never polled. A command without a run is handled by other code: the log window's key belongs to nvapp::ElementLogger. A binding with a text is matched by the code that handles it: '?' as a typed character by ui_help.cpp, and "Drop file", the modifiers held while a file is dropped, by onFileDrop(), which matches them against File > Import/Merge Scene and File > Reference Scene instead of naming the keys itself.

The viewport's mouse, wheel and movement keys are not commands: they are held down, dragged or clicked rather than pressed. They have a table of their own, kBindings in camera_controls.cpp; each row names an input, the modifiers it needs and those it tolerates, the navigation modes and projection it applies to, the action, and the label and tooltip Help shows. For an input, the first matching row wins. AnchorCameraManipulator dispatches drags and the wheel from it instead of using nvutils::CameraManipulator's own mapping, camera_controls::update() replaces nvapp::ElementCamera::updateCamera() for the viewport, mouseClickedInViewport() takes the click that selects and the double-click that sets the point of interest from it, and Help > Keyboard & Mouse draws its Camera tab (mouse grid, clicks and keys) from it. Rebinding them is a change to that table only.

A command that opens a dialog or rebuilds the scene only raises a request (m_menuRequests); renderMenu() carries it out once the menus are closed.

Add a renderer setting

  1. Add state in shared settings/resources.
  2. Bind and consume it in renderer_pathtracer.cpp or renderer_rasterizer.cpp.
  3. Expose UI controls in the corresponding UI module.
  4. Persist with the settings handler when needed.

Add or update glTF extension support

  1. Parse and serialize extension fields in tinygltf_utils.*.
  2. If the extension affects the material model (any KHR_materials_*), follow the checklist in Material System → Adding a new KHR_materials_* extension.
  3. Otherwise, integrate runtime behavior in scene/shader code as needed.
  4. If the extension stores any index (into accessors/bufferViews/textures or a root-level array it defines itself), add its remapping to gltf_scene_merger.cpp in the same change — otherwise it is silently dropped or cross-linked when scenes are merged. See External Assets → index remapping is exhaustive.
  5. Update extension documentation in README.md and usage notes in user-guide.md.

Modify sync/data-flow behavior

Use RENDERING_ARCHITECTURE.md as the source of truth for:

  • parse/rebuild flow
  • render-node dirty propagation
  • GPU buffer synchronization
  • TLAS/BLAS update invariants

Testing

Tests are disabled by default. Enable with BUILD_TESTING=ON:

cmake -B build -DBUILD_TESTING=ON
cmake --build build --target vk_gltf_renderer_tests
ctest --test-dir build -C Release --output-on-failure

See tests/README.md for details.

Contributing

This project uses the Developer Certificate of Origin (DCO). By contributing, you certify that you have the right to submit the work under the project's open source license.

Please open issues for bug reports and feature requests. Pull requests should target the main branch and include a clear description of the change.

Line endings

All text files are stored in the repository with LF. This is enforced by .gitattributes (* text=auto), with binary asset extensions pinned explicitly so they are never EOL-rewritten.

* text=auto normalizes to LF on commit but sets no working-tree eol attribute, so what you get on checkout is governed by your own Git configuration rather than by your platform:

Setting Working tree
core.autocrlf=true (Git for Windows default) CRLF
core.autocrlf=input LF
core.autocrlf=false follows core.eol, default native — CRLF on Windows, LF elsewhere
core.eol=lf LF, on any platform

You do not need to change whichever you use: every combination round-trips cleanly, so editing a file produces a diff of only the lines you actually changed.

The one-time renormalization that established this is listed in .git-blame-ignore-revs, so git blame skips it (GitHub and GitLab apply that file automatically).