---
title: Neural Six-way Lightmaps
url: https://www.emergentmind.com/topics/neural-six-way-lightmaps
type: topic
---

# Neural Six-way Lightmaps

Neural Six-way Lightmaps (NSLM) are a neural rendering technique designed for real-time depiction of participating media—such as smoke and volumetric light—within interactive virtual environments. NSLM fuses the efficiency of the traditional six-way lightmaps approach with deep neural inference to enable full user interactivity, including dynamic scene changes, camera and light movement, and obstacle interaction, at high visual fidelity and performance compatible with existing game engine pipelines. This method addresses the longstanding challenge of balancing computational efficiency, physical plausibility, and adaptability in volumetric rendering [2604.03748].

## 1. Background and Traditional Six-Way Lightmaps

Rendering participating media requires evaluating the volume rendering equation:
\[
L(x,\Omega_0) = \int_0^z T(x \leftrightarrow y) [\sigma_a(y) L_e(y, \Omega_0) + \sigma_s(y) L_s(y, \Omega_0)] dy + T(x \leftrightarrow z) L_0(z, \Omega_0)
\]
where $T(x\leftrightarrow y)$ is transmittance, $\sigma_a$ and $\sigma_s$ are absorption and scattering coefficients, and $L_s$ is the in-scattered radiance. 

The six-way lightmaps approximation precomputes, per frame and viewpoint, textures encoding transparency $\alpha(x)$ and six directional scattering maps $L_p^+, L_p^-$ for $p \in \{x,y,z\}$. Rendering then computes, for any light direction $\ell$, a weighted sum:
\[
L_\text{scatter}(x, \ell) \approx \sum_{p \in \{x, y, z\}} |\ell_p| \cdot L_p^{\text{sign}(\ell_p)}(x)
\]
These methods offer fast blending but are limited to static, pre-simulated sequences and fixed viewpoints, introducing artifacts and precluding interactivity [2604.03748].

## 2. Guiding Map Computation via Ray Marching

NSLM replaces full Monte Carlo integration with a fast guiding map $G(x) = (\hat{L}_\text{scat}, T, D)$ computed by single-pass, large-step ray marching along the camera direction $\Omega_c$. 

This map is derived using three fixed surrogate light directions:
- Front: $-\Omega_c$ (viewer-facing)
- Top: $+z$
- Bottom: $-z$

The approximate in-scattering is integrated as:
\[
\hat{L}_\text{scat}(x) = \int_0^z T(x\leftrightarrow y) \sigma_s(y) [T(y\leftrightarrow z_{front}) L_{front} + T(y\leftrightarrow z_{top}) L_{top} + T(y\leftrightarrow z_{bot}) L_{bot}] dy
\]
Discretization employs large steps ($h \approx 10\Delta x$). The output channels are $\hat{L}_\text{scat}$ (three-light in-scatter approximation), $T(x)$ (overall transparency), and $D(x)$ (depth to surface). This guide is intentionally coarse, reducing computational load and channel count for network inference [2604.03748].

## 3. Neural Network Architecture and Training

The NSLM network is a 2D image-to-image U-Net augmented with "channel adapters" to specialize outputs. The mapping is:
\[
\hat{L} = f_\theta(G), \quad \hat{L} \in \mathbb{R}^{H \times W \times 8}
\]
comprising six directional lightmaps, transparency, and an optional emissive channel.

- **Backbone:** U-Net, with encoder layers reducing to a bottleneck feature map followed by upsampling and skip connections.
- **Channel Adapters:** The eight output channels are grouped (front/back, left/right, up/down, {alpha, emissive}); each group is mapped with an adapter stack (2-3 NAFBlocks) for disentanglement.
- **Activation:** GELU or ReLU for all convs; sigmoidal or linear for the terminal layer.

**Training details:**
- Data generated via Lattice Boltzmann Method (LBM) simulations for 14 smoke+obstacle scenarios; 200 frames, 9 camera angles each ($\sim$25,200 frames).
- Ground-truth: Houdini Karma volume path tracer (512², 512 spp, single scatter, Henyey–Greenstein $g=0$).
- Loss: weighted combination of MSE (sRGB), VGG-perceptual, and FlowNet-based temporal consistency:
  \[
  \ell = \lambda_{MSE} \ell_{MSE} + \lambda_{perc} \ell_{perc} + \lambda_{flow} \ell_{flow}
  \]
  with typical weights 1.0, 0.1, 0.5, respectively.
- Optimization via Adam ($\beta_1=0.9, \beta_2=0.999$), learning rate $10^{-3}$, batch size 12, $\sim$200 epochs, $\sim$60 GPU hours, minimal regularization [2604.03748].

## 4. Pipeline Integration and Real-Time Shading

At inference, $f_\theta(G)$ generates two RGBA 512² textures:
- Tex0: (R,G,B,A) = ($+X$, $+Y$, $+Z$, transparency)
- Tex1: (R,G,B,A) = ($-X$, $-Y$, $-Z$, emissive)

Runtime shading per fragment:
1. Sample the six directional channels.
2. For incident light $\ell$, compute $L_{scat}$ as in the six-way formula.
3. Compute screen-space shadowing by comparing the smoke depth $D_{screen}$ with the light-view shadow map. Occluded samples have their $L_{scat}$ zeroed.
4. Final color $=$ $L_{scat}$ $+$ $\alpha$·emissive $+$ $(1-\alpha)$·background.

No geometric changes are required; standard billboard assets suffice, with updated textures and shader logic. This enables seamless integration into established pipelines [2604.03748].

## 5. Performance Benchmarks and Comparative Analysis

Experiments were conducted on an AMD Threadripper 3970X, NVIDIA GPU (16,384 cores, 24GB) at 512² resolution. Key timings per frame:

| Stage                        | Time (ms) |
|------------------------------|-----------|
| Fluid Simulation (100³ vel)  | 4.0       |
| Smoke Advection (400³ dens.) | 11.5      |
| Guiding Ray March (512²)     | 2.0       |
| Network Inference            | 1.2       |
| Shadow Test                  | 0.8       |
| Shader (blending)            | 0.3       |
| **Total (sim+render)**       | **19.8**  |

Comparative rendering metrics:

| Method                 | PSNR↑  | MSE↓     | Time (ms)        |
|------------------------|--------|----------|------------------|
| Reference Path-Tracer  | —      | —        | ~20,000          |
| ReSTIR (1 spp)         | 26.3   | 0.00109  | 10.4             |
| ReSTIR + denoiser      | 34.8   | 0.00032  | 12.6             |
| MRPNN (3D neural)      | 36.1   | 0.00030  | 4 + 180*         |
| **Neural 6-Way LM**    | 40.7   | 0.00008  | 3.9              |

*MRPNN: per-frame 3D feature precompute ≈ 180ms.

NSLM achieves significantly higher PSNR (38-41 dB on held-out sequences), lowest MSE, and sub-4 ms neural inference. The total rendering pipeline (simulation + rendering) remains under 20 ms per frame, supporting real-time workflows [2604.03748].

Memory requirements are low: two RGBA textures (8 MB, uncompressed) and network weights ($\sim$15 MB).

## 6. Qualitative Outcomes and Applications

Visual and perceptual benchmarks demonstrate:
- High temporal coherence (negligible flicker due to $\ell_{flow}$ loss).
- Robustness over a $\times$0.5–$\times$2.0 range of density scaling (PSNR $\geq$ 30 dB).
- Real-time support for dynamic scenarios, including smoke-geometry interactions, explosions with emissive components, and unrestricted camera or lighting changes in environments such as Unreal Engine.

Comparisons reveal that NSLM avoids the traditional billboard artifacts, surpasses ReSTIR in denoised quality, and outperforms volumetric 3D neural renderers in both fidelity and speed. A noted limitation is that screen-space shadowing, while efficient, can fail with complex internal occlusions [2604.03748].

## 7. Strengths, Limitations, and Prospects

Strengths:
- Full interactivity for smoke, obstacles, lighting, and camera movement.
- Real-time inference ($<$4 ms network, $<$20 ms end-to-end sim+render).
- High visual fidelity, capturing clear multiple-scatter effects.
- Plug-and-play compatibility with game engines.

Current limitations include reliance on screen-space approximations for shadowing (leading to artifacts for intricate occlusions), potential quality degradation on out-of-distribution density fields or highly complex geometries, and effective capture of only single-bounce scattering. 

Potential future advances involve learning more physically accurate volumetric shadowing (e.g., via screen-space virtual point lights), augmenting guiding maps with local gradients or additional cues for higher-order scattering, adaptive ray-marching for detail-sensitive regions, and direct inclusion of scene geometry in network conditioning for improved obstacle interactions. Further generalization to heterogeneous or Brownian media is also anticipated [2604.03748].

Source: https://www.emergentmind.com/topics/neural-six-way-lightmaps