Neural Six-Way Lightmaps

• The paper introduces a neural network framework that infers six directional lightmaps from coarse screen-space guiding maps, enabling real-time volumetric rendering of smoke.
• The approach integrates a U-Net architecture with specialized channel adapters to efficiently capture directional lighting cues and dynamic media variations.
• Results show a PSNR of 40.7 dB with a 3.9 ms lighting overhead, outperforming traditional methods while maintaining interactive performance.

Neural six-way lightmaps are a data-driven framework for real-time rendering of participating media, such as smoke, that synthesizes directional volumetric lighting with fidelity approaching path tracing while retaining the efficiency, interactivity, and integration of precomputed lightmap techniques. The approach leverages a neural network to infer six directional lightmaps from a coarse, view-aligned screen-space representation derived via approximate ray marching in the volume, enabling dynamic effects that support arbitrary camera movement, lighting changes, and real-time media simulation in interactive applications (Li et al., 4 Apr 2026).

1. Volume Rendering Foundations and Six-way Lightmaps

Volumetric participating media are governed by the volume rendering equation (VRE). For a point $\mathbf{x}$ and outgoing direction $\omega_o$, the VRE is

$$L(\mathbf{x},\omega_o) = \int_{0}^{d} T\bigl(\mathbf{x}\!\leftrightarrow\!\mathbf{y}(s)\bigr) \bigl[ \sigma_a(\mathbf{y}(s))\,L_e(\mathbf{y}(s),\omega_o) + \sigma_s(\mathbf{y}(s))\,L_s(\mathbf{y}(s),\omega_o) \bigr]\, ds + T\bigl(\mathbf{x}\!\leftrightarrow\!\mathbf{y}(d)\bigr)\, L_o(\mathbf{y}(d),\omega_o)$$

with $\sigma_a$, $\sigma_s$ as absorption and scattering coefficients, $T$ transmittance, $L_e$ emission, and $L_s$ in-scattered radiance (see data for full definitions). Direct Monte Carlo evaluation of the VRE is costly due to the need for many samples per pixel.

Conventional six-way lightmaps approximate volumetric effects by precomputing six 3D lightmaps aligned with the Cartesian axes (“front/back”, “left/right”, “up/down”), which are blended in real-time proportional to local lighting direction. This approach is widely adopted for games but is limited to pre-simulated sequences and static view assumptions, yielding inflexible results and suboptimal integration with dynamic, user-driven interactions.

2. Screen-space Guiding Map Generation

To enable a neural predictor to condition on dynamic volumetric geometry and view, a screen-space guiding map $\mathcal{G}(x)\in\mathbb{R}^3$ is computed per pixel $x$ through coarse, forward ray marching along the camera view direction $\omega_o$0. Key components calculated at each ray-marching step are:

• Transmittance:

$\omega_o$1

• Approximate in-scattered radiance: A weighted sum for primary axes (“front,” “top,” “bottom”) with large steps $\omega_o$2 and threshold-based accept/reject sampling.
• Layer Depth Estimation: Depth $\omega_o$3, indicating entry into the dense phase of the volume.

The resulting guiding map stacks $\omega_o$4, representing coarse 1st-order scattering, transmission, and silhouette structure.

3. Neural Network Architecture

Given the guiding map $\omega_o$5, a neural network models the mapping

$\omega_o$6

where each $\omega_o$7 approximates the light integral along cartesian directions. The architecture consists of:

• Input: $\omega_o$8 guiding map.
• Backbone: U-Net encoder–decoder, 4 down/up stages, 2×(Conv–ReLU–BatchNorm) per stage; bottleneck at $\omega_o$9.
• Channel Adapters: Four lateral heads (3 NAFBlocks each; 1×1 then 3×3 convolutions with SwiGLU) group outputs by axis pairs.
• Output: $$L(\mathbf{x},\omega_o) = \int_{0}^{d} T\bigl(\mathbf{x}\!\leftrightarrow\!\mathbf{y}(s)\bigr) \bigl[ \sigma_a(\mathbf{y}(s))\,L_e(\mathbf{y}(s),\omega_o) + \sigma_s(\mathbf{y}(s))\,L_s(\mathbf{y}(s),\omega_o) \bigr]\, ds + T\bigl(\mathbf{x}\!\leftrightarrow\!\mathbf{y}(d)\bigr)\, L_o(\mathbf{y}(d),\omega_o)$$0 tensor of six-way lightmaps.

Channel adapters enable the architecture to efficiently model correlations among the grouped axes and output features relevant to per-direction lighting synthesis.

4. Training Methodology and Loss Formulation

Training data is generated from 14 smoke sequences (400³ grid, 200 frames each). For each frame and 9 camera azimuths, high-fidelity ground-truth lightmaps $$L(\mathbf{x},\omega_o) = \int_{0}^{d} T\bigl(\mathbf{x}\!\leftrightarrow\!\mathbf{y}(s)\bigr) \bigl[ \sigma_a(\mathbf{y}(s))\,L_e(\mathbf{y}(s),\omega_o) + \sigma_s(\mathbf{y}(s))\,L_s(\mathbf{y}(s),\omega_o) \bigr]\, ds + T\bigl(\mathbf{x}\!\leftrightarrow\!\mathbf{y}(d)\bigr)\, L_o(\mathbf{y}(d),\omega_o)$$1 and guiding maps $$L(\mathbf{x},\omega_o) = \int_{0}^{d} T\bigl(\mathbf{x}\!\leftrightarrow\!\mathbf{y}(s)\bigr) \bigl[ \sigma_a(\mathbf{y}(s))\,L_e(\mathbf{y}(s),\omega_o) + \sigma_s(\mathbf{y}(s))\,L_s(\mathbf{y}(s),\omega_o) \bigr]\, ds + T\bigl(\mathbf{x}\!\leftrightarrow\!\mathbf{y}(d)\bigr)\, L_o(\mathbf{y}(d),\omega_o)$$2 are rendered in Houdini Karma using Monte Carlo (512 spp, single bounce). The resulting dataset comprises 25,200 examples.

The network is trained to minimize a compound loss: $$L(\mathbf{x},\omega_o) = \int_{0}^{d} T\bigl(\mathbf{x}\!\leftrightarrow\!\mathbf{y}(s)\bigr) \bigl[ \sigma_a(\mathbf{y}(s))\,L_e(\mathbf{y}(s),\omega_o) + \sigma_s(\mathbf{y}(s))\,L_s(\mathbf{y}(s),\omega_o) \bigr]\, ds + T\bigl(\mathbf{x}\!\leftrightarrow\!\mathbf{y}(d)\bigr)\, L_o(\mathbf{y}(d),\omega_o)$$3 where $$L(\mathbf{x},\omega_o) = \int_{0}^{d} T\bigl(\mathbf{x}\!\leftrightarrow\!\mathbf{y}(s)\bigr) \bigl[ \sigma_a(\mathbf{y}(s))\,L_e(\mathbf{y}(s),\omega_o) + \sigma_s(\mathbf{y}(s))\,L_s(\mathbf{y}(s),\omega_o) \bigr]\, ds + T\bigl(\mathbf{x}\!\leftrightarrow\!\mathbf{y}(d)\bigr)\, L_o(\mathbf{y}(d),\omega_o)$$4 denotes VGG feature extraction for perceptual similarity and the FlowNet term promotes temporal coherence. Coefficients are $$L(\mathbf{x},\omega_o) = \int_{0}^{d} T\bigl(\mathbf{x}\!\leftrightarrow\!\mathbf{y}(s)\bigr) \bigl[ \sigma_a(\mathbf{y}(s))\,L_e(\mathbf{y}(s),\omega_o) + \sigma_s(\mathbf{y}(s))\,L_s(\mathbf{y}(s),\omega_o) \bigr]\, ds + T\bigl(\mathbf{x}\!\leftrightarrow\!\mathbf{y}(d)\bigr)\, L_o(\mathbf{y}(d),\omega_o)$$5, $$L(\mathbf{x},\omega_o) = \int_{0}^{d} T\bigl(\mathbf{x}\!\leftrightarrow\!\mathbf{y}(s)\bigr) \bigl[ \sigma_a(\mathbf{y}(s))\,L_e(\mathbf{y}(s),\omega_o) + \sigma_s(\mathbf{y}(s))\,L_s(\mathbf{y}(s),\omega_o) \bigr]\, ds + T\bigl(\mathbf{x}\!\leftrightarrow\!\mathbf{y}(d)\bigr)\, L_o(\mathbf{y}(d),\omega_o)$$6. Optimization uses Adam (lr $$L(\mathbf{x},\omega_o) = \int_{0}^{d} T\bigl(\mathbf{x}\!\leftrightarrow\!\mathbf{y}(s)\bigr) \bigl[ \sigma_a(\mathbf{y}(s))\,L_e(\mathbf{y}(s),\omega_o) + \sigma_s(\mathbf{y}(s))\,L_s(\mathbf{y}(s),\omega_o) \bigr]\, ds + T\bigl(\mathbf{x}\!\leftrightarrow\!\mathbf{y}(d)\bigr)\, L_o(\mathbf{y}(d),\omega_o)$$7, batch 12, 200 epochs, $$L(\mathbf{x},\omega_o) = \int_{0}^{d} T\bigl(\mathbf{x}\!\leftrightarrow\!\mathbf{y}(s)\bigr) \bigl[ \sigma_a(\mathbf{y}(s))\,L_e(\mathbf{y}(s),\omega_o) + \sigma_s(\mathbf{y}(s))\,L_s(\mathbf{y}(s),\omega_o) \bigr]\, ds + T\bigl(\mathbf{x}\!\leftrightarrow\!\mathbf{y}(d)\bigr)\, L_o(\mathbf{y}(d),\omega_o)$$860 hours on a single GPU).

5. Pipeline Integration and Rendering Process

At runtime, the pipeline executes the following stages:

1. Simulation: 3D LBM velocity (4 ms) and density advection (11.5 ms).
2. Guiding Map Generation: Coarse ray march at $$L(\mathbf{x},\omega_o) = \int_{0}^{d} T\bigl(\mathbf{x}\!\leftrightarrow\!\mathbf{y}(s)\bigr) \bigl[ \sigma_a(\mathbf{y}(s))\,L_e(\mathbf{y}(s),\omega_o) + \sigma_s(\mathbf{y}(s))\,L_s(\mathbf{y}(s),\omega_o) \bigr]\, ds + T\bigl(\mathbf{x}\!\leftrightarrow\!\mathbf{y}(d)\bigr)\, L_o(\mathbf{y}(d),\omega_o)$$9 (2 ms).
3. Neural Inference: TensorRT-accelerated U-Net evaluation (1.2 ms).
4. Shading: Camera-facing billboard samples $\sigma_a$0 at each fragment (0.3 ms).
5. Shadowing: Screen-space depth test compares guiding map depth $\sigma_a$1 to obstacle shadow map in light space (0.8 ms).

The resulting pixel color under arbitrary light direction $\sigma_a$2 is computed as: $\sigma_a$3 where $\sigma_a$4 are the directional lightmaps, $\sigma_a$5 is predicted transparency, and $\sigma_a$6 is an optional emission component.

6. Benchmarks: Performance, Fidelity, and Comparisons

Comparisons to existing methods demonstrate the efficiency and visual quality of neural six-way lightmaps:

Method	Real-time	PSNR (↑)	Time (ms ↓)	Memory
Traditional 6-way [Müller]	yes	27.5 dB	0.5	2×512² × RGB
Volumetric ReSTIR (1 spp)	no	26.3 dB	10.4	–
ReSTIR + denoise	no	34.8 dB	12.6	–
MRPNN	no	36.1 dB	184.0	–
Ours (neural 6-way)	yes	40.7 dB	3.9	512² × 6 ch

Key metrics:

• Full pipeline runs at $\sigma_a$720 ms per frame (50 FPS), with 3.9 ms attributed to lighting.
• Visual fidelity measured by PSNR: +4 dB over denoised ReSTIR, +4.6 dB over traditional lightmaps.
• Memory footprint: 6-channel lightmap per pixel.

7. Limitations and Future Research Directions

Limitations include:

• Shadow singularities: Screen-space depth test yields under-shadowing beneath complex geometry due to absence of complete volumetric shadow propagation.
• Distribution shift: For smoke densities scaled by $\sigma_a$8 or $\sigma_a$9, PSNR degrades by $\sigma_s$0 dB but plausibility is preserved.

Identified avenues for future research:

• Integration of higher-order scattering via learned residual corrections.
• Generalization to area light sources or environment lighting.
• Improved dynamic geometry shadowing through volumetric carving in light space.
• Extension to other participating media (clouds, fire with anisotropic phase functions) (Li et al., 4 Apr 2026).

Neural six-way lightmaps thus provide an efficient and scalable solution for realistic, interactive volumetric rendering in dynamic virtual environments, bridging the gap between traditional precomputed lightmaps and computationally intensive path-traced methods.