Channel-aware Neural Lightmap Prediction

• The paper introduces explicit color-channel modeling using spherical harmonics, RGBA grids, and six-way lightmaps to accurately estimate spatially varying lighting.
• It utilizes multi-stream CNN architectures and channel-adapted decoders that fuse global and local features to preserve chromatic details in real-time performance.
• Empirical evaluations demonstrate strong metrics such as MAE and PSNR, validating the approach for applications in indoor relighting, AR, and real-time volumetric effects.

Channel-aware neural lightmap prediction encompasses a family of deep learning-based techniques for inferring spatially-varying lighting, often with explicit color-channel modeling, from visual input such as monocular or stereo imagery. These methods are characterized by explicit representations of lighting that preserve per-channel (e.g., RGB) color, enabling applications ranging from photorealistic relighting to real-time volumetric effects in graphics and vision. Across the domain, key architectural and representational innovations include per-channel spherical harmonics, volumetric RGBA grids, and channel-adaptive neural decoders. Channel-awareness ensures that color fidelity and chromatic illumination cues are preserved throughout inference and rendering.

1. Mathematical Representations for Channel-aware Lightmaps

Channel-aware neural prediction architectures typically model incident radiance at each scene location as a quantity with distinct color-channel dependencies. Three principal formulations are prevalent:

1. Spherical Harmonics (SH) Lighting Representation Incident radiance at a point is projected onto real SH basis functions $Y_l^m(\omega)$. For RGB lighting, the estimated radiance in direction $\omega$ is:

$$L(\omega) = \sum_{l=0}^L \sum_{m=-l}^l c_{l\,m}^r Y_l^m(\omega) \,\hat{\mathbf e}_r + c_{l\,m}^g Y_l^m(\omega)\,\hat{\mathbf e}_g + c_{l\,m}^b Y_l^m(\omega)\,\hat{\mathbf e}_b$$

Each channel $c \in \{r, g, b\}$ has $(L+1)^2$ coefficients; for $L=5$ as in (Garon et al., 2019), the prediction head outputs $108$ SH coefficients, explicitly governed by color channel.

2. Volumetric RGBA Lighting Volumes The Lighthouse framework (Srinivasan et al., 2020) employs multiscale 3D RGBA grids. Each voxel value $V(x) \in \mathbb{R}^4$ encodes predicted color radiance $(c_r, c_g, c_b)$ and opacity $\alpha$ at location $\omega$0:

$\omega$1

Environment maps or directional light probes are rendered via volumetric alpha composition, ensuring channel consistency.

3. Six-way Lightmaps for Participating Media For dynamic volumetric effects, as in real-time neural six-way lightmaps (Li et al., 4 Apr 2026), the predicted output is a set of scattering lightmaps $\omega$2, as well as transmittance $\omega$3 and (optionally) emission $\omega$4. The neural head is constructed to predict these channel bundles, with adapters splitting output features by lightmap direction and effect.

2. Neural Architectures and Channel-aware Decoding

Distinct channel-aware architectures are utilized to fuse spatial context, enforce color-differentiated lighting, and enable real-time performance.

• Two-stream CNNs with Feature Fusion (Garon et al., 2019) The network comprises global (full image) and local (patch) feature paths. Global features, augmented with a binary position mask, are extracted via DenseNet-121 backbones, while local context is processed independently. Fused features drive heads for lighting (explicitly tri-channel SH coefficient outputs), depth SH, and albedo/shading. Explicit separation of RGB SH blocks in the lighting head enforces channel-awareness at inference.
• Channel-adapted Decoders in Volumetric and Billboard Rendering (Srinivasan et al., 2020, Li et al., 4 Apr 2026) RGBA volume prediction in Lighthouse employs 3D U-Nets with per-voxel multichannel outputs (3 for color, 1 for opacity). For six-way lightmaps, the network's decoder is divided into channel adapters, channel-specialized residual blocks that yield the six scattering and two auxiliary (transmittance, emission) planes, each respecting physical structure and channel separation. Channel-wise feature gating and ReLU activations maintain color specificity.

3. Training Paradigms, Supervision, and Loss Composition

Channel-aware neural lightmap prediction relies on self-consistent, data-driven supervision, typically using synthetic datasets with ground-truth lighting, depths, and/or volume renderings:

• Synthetic Cubemap and SH Fitting (Garon et al., 2019) Lighting ground-truth is derived by path-tracing cubemaps at probe locations, followed by SH projection. Losses comprise MSE over all 108 channel-SH coefficients, depth-SH regression, and pixelwise MSE for albedo/shading, with a multitask framework enhancing invariance and accuracy. Domain adaptation (via cross-entropy and a Gradient Reversal Layer) enables bridging to real-captured data.
• 3D RGBA Volume Supervision and Differentiable Volume Rendering (Srinivasan et al., 2020) The Lighthouse system uses stereo synthetic or photorealistic pairs for scene input, with supervision via held-out renderings and environment panoramas. Perceptual losses on renderings, plus adversarial training for finer lighting structure, drive channel-consistent learning. The volume architecture, through alpha compositing, inherently propagates channel distinctions.
• Six-way Lightmap Regression with Spatiotemporal Losses (Li et al., 4 Apr 2026) Channel-adaptive six-way lightmaps are trained with MSE on reference path-traced lightmap textures, VGG-based perceptual losses, and optical-flow-based temporal stability terms. Ablation studies demonstrate that both channel-adapter decoder splitting and the inclusion of perceptual/flow losses are critical for detail preservation and chromatic consistency.

4. Quantitative Performance and Comparative Evaluation

Channel-aware neural lightmap prediction exhibits strong quantitative gains and efficiency:

Method (Paper)	Output Representation	Accuracy/Metric	Runtime (per frame)
SH-CNN (Garon et al., 2019)	$\omega$5-order SH (RGB)	MAE/Root-MSE vs GT: $\omega$6	20 ms (GTX 970M)
Lighthouse (Srinivasan et al., 2020)	RGBA Volumetric Grid	PSNR: $\omega$7 dB / Angular error: $\omega$8	Not specified
Neural Six-way (Li et al., 4 Apr 2026)	Six lightmaps + aux	PSNR: $\omega$9 dB (best, front-top-bottom)	$$L(\omega) = \sum_{l=0}^L \sum_{m=-l}^l c_{l\,m}^r Y_l^m(\omega) \,\hat{\mathbf e}_r + c_{l\,m}^g Y_l^m(\omega)\,\hat{\mathbf e}_g + c_{l\,m}^b Y_l^m(\omega)\,\hat{\mathbf e}_b$$04 ms (512x512)

Performance ablations confirm the value of channel-aware head structuring and multitask settings: for SH-CNNs, combining global and local context outperforms either alone; the addition of depth and albedo/shading heads differentially improves various SH degree errors. In six-way systems, channel adapters yield significant PSNR gains ($$L(\omega) = \sum_{l=0}^L \sum_{m=-l}^l c_{l\,m}^r Y_l^m(\omega) \,\hat{\mathbf e}_r + c_{l\,m}^g Y_l^m(\omega)\,\hat{\mathbf e}_g + c_{l\,m}^b Y_l^m(\omega)\,\hat{\mathbf e}_b$$1–$$L(\omega) = \sum_{l=0}^L \sum_{m=-l}^l c_{l\,m}^r Y_l^m(\omega) \,\hat{\mathbf e}_r + c_{l\,m}^g Y_l^m(\omega)\,\hat{\mathbf e}_g + c_{l\,m}^b Y_l^m(\omega)\,\hat{\mathbf e}_b$$2 dB). In user studies, confusion rates for channel-aware SH lightmap estimation approach 36% versus ideal 50% (ground truth indistinguishable).

5. Applications in Graphics, Vision, and AR

Channel-aware neural lightmap prediction has been adopted for:

• Indoor relighting and augmented reality: Inserting and relighting virtual objects with scene-consistent illumination, utilizing fast per-location RGB lighting estimates without geometry or HDR supervision (Garon et al., 2019).
• Volumetric effects for real-time graphics: Dynamic, light-consistent rendering of smoke and participating media in games/VR/AR, achieved by neural six-way lightmaps that replicate classical flipbook shading while supporting interaction, view/lighting variation, and runtime execution (Li et al., 4 Apr 2026).
• Photorealistic scene relighting and insertion: The RGBA lighting volume approach of Lighthouse enables insertion of objects in arbitrary 3D locations, where the channel-consistent volumetric field supports high-specular and spatially coherent lighting (Srinivasan et al., 2020).

6. Limitations, Ablation Insights, and Future Directions

Recognized limitations motivate further research:

• Channel-aware predictors, while robust to moderate albedo variation, are susceptible to domain shifts—e.g., hue-shifts occur when color space between training and real data diverges (Garon et al., 2019).
• In volumetric (six-way) models, screen-space or shell-based guiding maps constrain the system’s ability to model deep, mid-volume self-shadowing; hidden volume variations induce generalization gaps (Li et al., 4 Apr 2026).
• Multi-scale volume completion in Lighthouse depends on coherent hallucination of unseen content; ablations reveal pronounced quality drops if only observed voxels are passed directly to rendering (Srinivasan et al., 2020).

Emergent avenues include extending channel-aware lightmap networks to additional participating media (e.g., clouds, fire), integrating learned depth-volume shadowing, and exploring channel-specific normalization or regularization for improved out-of-distribution color fidelity.

7. Key Contributions and Summary

The core advances of channel-aware neural lightmap prediction are:

1. Explicit color-channel modeling that preserves spectral composition in inferred illumination.
2. Architectures enabling both local detail and global context fusion, via parallel CNN streams, volumetric grids, or U-Net with channel adapters.
3. Multitask and perceptual supervision that regularizes channel fidelity while supporting domain adaptation.
4. Empirically validated real-time performance with competitive accuracy and perceptual quality over prior global or monochrome estimators across a range of graphics and vision scenarios.

These methods collectively define the state of the art for deep color-consistent spatially-varying lighting estimation and neural-based lightmap prediction (Garon et al., 2019, Srinivasan et al., 2020, Li et al., 4 Apr 2026).