Spherical Harmonics Relighting Pipeline

• Spherical Harmonics-based relighting pipeline is a framework that encodes incident lighting into compact SH coefficients to simulate diffuse and low-frequency specular illumination.
• It integrates mathematical expansion, deep learning regression, and shader evaluation to achieve real-time and high-fidelity global illumination across various devices.
• Despite its efficiency, the technique faces limitations with high-frequency details and occlusion effects, often addressed through hybrid models and advanced optimization strategies.

Spherical Harmonics-based Relighting Pipeline

A spherical harmonics (SH)–based relighting pipeline is an efficient framework for capturing and reproducing incident lighting in a scene for realistic illumination synthesis and augmented reality applications. The principal goal is to regress or encode the incident radiance function on the sphere surrounding a spatial location, then use this compact representation for high-fidelity, real-time or offline rendering. SH-based methodologies benefit from the orthonormality and rotational properties of spherical harmonics, enabling accurate simulation of diffuse and low-frequency specular lighting at minimal computational cost. Contemporary relighting pipelines further leverage point-cloud deep learning, hybrid parametric models, physically-based transport, and differentiable solvers to achieve high visual quality on both mobile and desktop platforms.

1. Mathematical Framework of Spherical Harmonics Relighting

At the core of SH relighting is the expansion of the incident radiance $L(\theta, \phi)$ as a truncated sum of real-valued spherical harmonics $Y_{\ell m}$ up to degree $\ell$. For $\ell \leq 2$ (second order, 9 basis functions per channel):

$$L(\theta, \phi) = \sum_{\ell=0}^{2} \sum_{m=-\ell}^{\ell} c_{\ell m} Y_{\ell m}(\theta, \phi)$$

where $c_{\ell m}$ are the SH coefficients capturing projection of $L$ onto each basis function (Zhao et al., 2020). For computational efficiency, algorithms approximate lighting integrals via Monte Carlo sampling or closed-form convolution, with well-known coefficients for Lambertian surfaces (Ramamoorthi & Hanrahan 2001).

The SH basis facilitates efficient per-pixel irradiance computation:

$$E_c(\mathbf{n}) = \sum_{\ell=0}^{2} \sum_{m=-\ell}^{\ell} c_{\ell m, c} Y_{\ell m}(\theta_{\mathbf{n}}, \phi_{\mathbf{n}})$$

where $\mathbf{n}$ is the surface normal, and $c$ indexes RGB channels.

2. Estimation and Regression of SH Coefficients

A diversity of pipelines exist for estimating the SH coefficients for local illumination, often from sensor data (e.g., RGB-D images, photographs, or panoramas):

• PointAR Pipeline: Uses a two-stage pipeline consisting of (1) geometry-aware transformation to extract a local recentered point cloud from an RGB-D frame and (2) regression of SH coefficients directly from point cloud samples via a compact PointConv network, learning a data-driven aggregation that replaces classic Monte Carlo integration. The network outputs $S_r \in \mathbb{R}^{3\times9}$ SH coefficients for each RGB channel at the user-chosen location (Zhao et al., 2020).
• MixLight Architecture: Proposes a joint regression model mapping an RGB crop to both (a) SH coefficients (ambient/low-frequency lighting) and (b) parameters of spherical Gaussians (SGs, for high-frequency light sources). The unique SLSparsemax module enforces sparsity and local clustering for SG lobes, while a suite of losses controls both coefficient fidelity and rendered shading accuracy. The SH head outputs 27 parameters (3 channels × 9 bases) for ambient light (Ji et al., 2024).
• Occlusion-aware Human Relighting: Uses convolutional neural networks (CNNs) to simultaneously infer: (a) diffuse albedo, (b) SH lighting, and (c) a per-pixel SH transport map encoding both direct and occluded visibility. Ground truth SH transport vectors for supervised training are computed via physically-based path tracing (Kanamori et al., 2019).
• Self-supervised Siamese Autoencoder: Enforces the latent lighting embedding to match SH transform properties under image augmentation, enabling robust estimation of scene illumination from unconstrained images using only self-supervision. This approach yields a 9-dimensional latent SH code per image, facilitating realistic relighting even in the absence of explicit geometry (Liu et al., 2020).

3. Integration with Rendering Pipelines

Once SH coefficients are regressed or encoded:

• Shader Evaluation: Renderer shaders evaluate per-pixel irradiance using the regressed SH coefficients, typically exploiting the orthonormality and closed forms for $Y_{\ell m}$. Constants can be precomputed to multiply normal components, reducing the computational load to a series of multiplies and adds, suitable for mobile GPUs (Zhao et al., 2020).
• Hybrid SH+SG Representations: For environments with both diffuse ambient and sharp directional light sources, hybrid pipelines combine SH (for low-frequency ambient) and SG (for high-frequency peaks). The HDR environment is split; peaks are fit by SGs, and the residual projected onto SH. Rendering accumulates contributions from both bases, supporting non-Lambertian surfaces and richer lighting interactions (e.g., object insertion in NeRF-based AR) (Ji et al., 2024, Zhu et al., 2024).
• Global Illumination and Baking: Static GI pipelines such as WishGI perform offline least-squares fitting of sampled radiance into vertex-centric SH probes, with blue-noise distribution and geodesic clustering ensuring efficient association and high coverage. At runtime, vertex shaders reconstruct global illumination by combining SH coefficients from associated probes and evaluating the spherical harmonic sum along view directions (Zhu et al., 2 Jun 2025).

4. Physically-Based Light Transport and Radiosity in SH Domain

SH relighting enables efficient simulation of physically-based light transport by encoding radiance fields and transport operators in the coefficient space:

• Gaussian Surfel Radiosity: The scene is represented as a set of Gaussian surfels, each carrying SH-expansion coefficients for both emission and outgoing radiance. Forward global illumination is formulated as a radiosity system in SH space, with non-binary visibility and semi-opaque modeling. Efficient solvers (progressive refinement and Monte Carlo, hybridized for speed and variance reduction) solve for the steady-state SH radiance. The pipeline supports analytic backward passes for gradient optimization, enabling real-time relighting and material editing at hundreds of FPS (Jiang et al., 23 Sep 2025).
• Precomputed Radiance Transfer (PRT) with 3D Gaussians: Gaussian spats are assigned SH transfer vectors via point-location–dependent neural decoders. Free-viewpoint relighting is achieved through dot-product evaluation of the projected environment SH coefficients and learned per-Gaussian SH transport (Zhang et al., 2024). This yields an efficient framework ideal for real-time rendering and inverse rendering applications.

5. Deep Learning and Differentiable SH Relighting

Modern SH-based relighting frameworks increasingly leverage deep learning and differentiable optimization:

• Network-based Inverse Rendering: Joint estimation of geometry, reflectance, and illumination is achieved via multi-head CNN or U-Net architectures, with smoothness and reconstruction constraints coupled to differentiable rendering layers. Loss terms span albedo, transport, illumination, image reconstruction, and total-variation regularization (Kanamori et al., 2019, Yi et al., 2023, Lagunas et al., 2021).
• Differentiable Layer Implementation: Both Lambertian and low-frequency specular shading terms are implemented as closed-form differentiable tensor operations, supporting automatic gradient flow throughout the pipeline (e.g., in PyTorch) (Yi et al., 2023).
• Self-supervised Constraints: Transform consistency of SH codes under known image augmentation operations is enforced by SH-loss functions, enabling unsupervised disentanglement of content and illumination (Liu et al., 2020).

6. Efficiency, Mobile Applicability, and Performance Benchmarks

SH-based pipelines excel in resource efficiency and real-time applicability:

• Mobile Complexity: PointAR achieves order-of-magnitude lower resource perplexity compared to image-based deep models, with sub-10 ms latency and 30 FPS updates on mobile GPUs, attributed to compact PointConv architectures and leveraged point cloud sampling (Zhao et al., 2020).
• Low-memory GI Baking: WishGI reduces memory to 5% of mainstream techniques for static GI by compressing illumination into SH probe coefficients. For typical meshes, only 0.6–1.0 KB is needed, achieving 36.7 FPS on flagship Snapdragon devices (Zhu et al., 2 Jun 2025).
• Real-time Light Rotation and Dynamic Scenarios: Neural-GASh in Unity exploits CGA algebra for vertex feature encoding and on-the-fly SH rotation, achieving rendering speeds competitive with or exceeding offline PRT even for animated or deformed meshes (Geronikolakis et al., 18 Jul 2025).
• Quality Metrics: Quantitative benchmarks across pipelines measure $L_2$ loss on SH coefficients, reconstructed irradiance loss, RMSE on rendered spheres, and standard image similarity metrics (PSNR, SSIM, LPIPS). SH pipelines consistently outperform prior image or GAN-based models in both controlled and in-the-wild environments (Zhao et al., 2020, Ji et al., 2024, Liu et al., 2020).

7. Limitations and Extensions

Despite their strengths, SH-based pipelines have intrinsic limitations:

• Low-frequency Approximation: SH expansions are fundamentally limited to low-frequency signals; sharp highlights or small, bright sources require hybridization with SG kernels or explicit transport (as in MixLight or hybrid NeRF pipelines) (Ji et al., 2024, Zhu et al., 2024).
• Occlusion and High-frequency Effects: Capturing complex occlusion or view-dependent specular lobes with SH alone can lead to overly smooth shading and penumbra artifacts. Solutions include learned transport maps, residual term decomposition, or increased SH band count (at a computational cost) (Kanamori et al., 2019, Lagunas et al., 2021, Silvestre et al., 2024).
• Parametric Model Selection: The number and positioning of SG lobes in hybrid models is often hand-selected and may require empirical tuning for complex scenes (Ji et al., 2024, Zhu et al., 2024).

Advancements involve higher-order expansions, more expressive hybrid models, improved sparsity-enforcing modules, physically-based inverse transport solvers, and differentiable global illumination frameworks, ensuring ongoing research for realistic and efficient relighting across dynamic and static content.