---
title: Relightable 3D Gaussian Splatting
url: https://www.emergentmind.com/topics/relightable-3d-gaussian-splatting-representation
type: topic
---

# Relightable 3D Gaussian Splatting

A relightable 3D Gaussian Splatting representation is an image- and light-field adaptative rendering model that encodes both geometry and physically based appearance with anisotropic Gaussian primitives, enabling photorealistic, real-time relighting and novel-view synthesis for complex scenes and assets. Unlike classical mesh or volumetric models, this approach employs a sparse cloud of learnable Gaussian ellipsoids, each parameterized for geometry, material, and radiance transfer. By combining explicit geometry fitting, precomputed radiance transfer, deferred shading, and—where necessary—bidirectional spherical harmonics or neural reflectance models, these methods unify the efficiency of 3D Gaussian splatting with accurate lighting simulation and material decomposition. Recent advances enable not only high-fidelity head avatars able to resolve hair/facial microgeometry and eye glints but also scalable scene-level representations that support shadows, interreflection, global illumination, and material editing across indoor and outdoor domains.

## 1. Mathematical Foundations and Primitive Parameterization

A relightable 3D Gaussian primitive is defined by:

- **Center:** $\mu_k \in \mathbb{R}^3$
- **Covariance:** $\Sigma_k = R_k \,\text{diag}(s_k)^2\,R_k^\top$, with $R_k \in SO(3)$, $s_k \in \mathbb{R}^3$ per-axis scales
- **Opacity:** $o_k \in [0,1]$
- **Material / Appearance:** Including base albedo $\rho_k \in \mathbb{R}^3$, roughness $r_k$, metalness $m_k$, radiance transfer coefficients, and (optionally) view/light-dependent SH, SG, or neural-MLP features

For a point $x \in \mathbb{R}^3$, the unnormalized Gaussian density is:
\[
g_k(x) = w_k \exp\left(-\tfrac{1}{2} (x - \mu_k)^\top \Sigma_k^{-1} (x - \mu_k)\right)
\]

Projection to the image plane uses the combined camera pose and Jacobian: the projected Gaussian center is $\mu_k'$, and the covariance is $\Sigma_k' = J V \Sigma_k V^\top J^\top$. The per-pixel color $C_p$ is composited front-to-back:
\[
C_p = \sum_{k=1}^{N} c_k \,\alpha_k\,\prod_{j<k}(1-\alpha_j)
\]
where $\alpha_k$ is the screen-space opacity determined by the projected area and the learned $o_k$.

Material attributes per Gaussian vary with the method but commonly include explicit physically based rendering (PBR) parameters (albedo, roughness, metallicity) and radiance transfer coefficients (SH for diffuse/global effects, SG or angular Gaussians for higher-frequency/lobe effects) [2312.03704, 2409.10335, 2408.03538].

## 2. Appearance and Relighting Models

To support accurate relighting, 3DGS representations decompose outgoing radiance into multiple components per Gaussian:

- **Diffuse Component:** Captures view-independent energy, often parameterized using spherical harmonics (SH), representing both the illumination and the intrinsic per-Gaussian transfer function (occlusion, multi-bounce, subsurface scattering).
  \[
  c_k^\text{diffuse} = \rho_k \circ \sum_{i=1}^{(n+1)^2} L_i \circ d_k^i
  \]
  where $L_i$ are SH lighting coefficients and $d_k^i$ are per-Gaussian transfer coefficients [2312.03704, 2408.03538].

- **Specular Component:** Modeled either by spherical Gaussian (SG) lobes aligned with the reflection axis, with learned width/visibility, or, in more expressive settings, as a mixture of anisotropic angular Gaussians or by bidirectional SH for full view- and light-dependence [2312.03704, 2410.11419]. For SG:
  \[
  c_k^\text{specular}(\omega_o) = v_k(\omega_o) \int_{S^2} L(\omega_i) G_s(\omega_i; q_k, \sigma_k) d\omega_i
  \]
  where $q_k$ is the reflection axis, $v_k$ is a visibility mask, and $G_s(\cdot)$ is the normalized SG kernel.

- **Radiance Transfer Functions:** For low-frequency environments or diffuse-dominant scenes, precomputed radiance transfer (PRT) expands the transfer kernel in SH basis:
  \[
  L_o(x) = \sum_{j=1}^N \ell_j t_j(x)
  \]
  or, for glossy surfaces, an SH matrix that enables efficient dot-product evaluation at runtime [2408.03538].

- **Learned Neural Corrections:** For domains with intricate subsurface, non-Lambertian, or volumetric effects, small neural networks (e.g., MLPs) are employed to regress the diffuse or global-illumination residuals as functions of light, view, and local Gaussian parameters [2411.12510, 2410.11419].

## 3. Deferred Shading and Rendering Pipelines

Relighting within the 3DGS paradigm can be performed through either alpha compositing (classical splatting) or deferred shading pipelines:

- **Standard Splatting:** Gaussians are rasterized in screen (or light) space by computing their projected 2D ellipses, followed by ordered front-to-back blending of their appearance, weighted by per-Gaussian opacity and (optionally) precomputed shadow or interreflection terms [2312.03704, 2509.11275].

- **Deferred Rendering:** Separates rasterization into a G-buffer stage, where all per-pixel geometric and material information is composited, and a screen-space shading pass that evaluates the BRDF under the current illumination [2409.10335, 2504.01358, 2507.15629]. This eliminates hidden-surface artifacts where Gaussians beneath the visible surface impact the result.

- **Ray-Tracing Integration:** Some frameworks (e.g., RaySplats) merge Gaussians and triangle meshes in a BVH, allowing full ray-tracing for shadows, interreflection, and refraction, with differentiable index buffers supporting efficient backward passes [2501.19196].

- **Triple Splatting:** GS³ performs three passes—camera-space for shading, light-space for shadow accumulation, and a global-illumination pass—followed by compositing. Shadows and GI are refined via small neural networks conditioned on local Gaussian features [2410.11419].

## 4. Learning Strategies and Regularization

Effective relightable 3DGS requires robust geometry, material-light decomposition, and stability. Approaches include:

- **Geometry refinement:** Using monocular normal priors, mesh extraction, or SDF-based projection constraints, methods regularize Gaussian placement to avoid blobby or misaligned surfaces. COREA employs dual-branch bidirectional supervision aligning SDF and Gs splat-cloud geometry [2507.15629, 2512.07107].
- **Material decomposition:** Where possible, SH/PRT coefficients and per-Gaussian BRDF parameters are learned with multi-stage schedules, often starting with diffuse-only or flat-lit appearance and progressing to full view/light parameterization [2601.03357, 2312.03704].
- **Opacity/scale regularization:** Controls over opaque region coverage and per-Gaussian size keep the representation compact and conformant to surface shape [2409.10335].
- **Losses:** Image-space L₁, SSIM, and photometric metrics dominate, but normal/curvature, opacity-masked, SDF-consistency, and radiance/SH smoothness losses provide crucial stabilization and disentanglement [2409.10335, 2507.15629, 2512.07107].

## 5. Domain-Specific Adaptations and Extensions

Relightable 3DGS frameworks have been generalized for:

- **Head avatars and avatars with explicit dynamic expressivity:** e.g., Relightable Gaussian Codec Avatars and RelightAnyone use expressive geometry+SH+SG parameterizations, CVAEs for animation, and explicit analytic eye models for high-fidelity glints and gaze control [2312.03704, 2601.03357].
- **General assets and outdoor/indoor scenes:** Adaptations to support sun/sky/indirect-light decomposition (e.g., ROSGS, GaRe), hybrid mesh+GS representations for precise occlusion, and large-scale asset pipelines with PBR material prediction (e.g., MGM, GRGS) [2509.11275, 2505.21502, 2509.22112].
- **Medical imaging:** PR-ENDO customizes the appearance model to endoscopy, employing a diffuse MLP and camera-light coupled physically-based rendering [2411.12510].
- **Volumetric/anisotropic/unstructured materials:** BiGS and GS³ drop or generalize normals and incorporate bidirectional SH or angular-Gaussian scattering for fluffy/fur/translucent objects [2408.13370, 2410.11419].

## 6. Quantitative Performance, Applications, and Limitations

Most 3DGS relightable frameworks match or surpass previous real-time relighting and inverse-rendering methods in PSNR/SSIM/LPIPS, with real-time rendering rates (30–100+ fps) and training times ranging from less than an hour for scene reconstructions to a few hours for high-fidelity avatars [2312.03704, 2410.11419, 2408.03538]. Explicit support for dynamic lighting, shadows, and view-dependent effects enables real-time VR avatars, scene relighting/editing, object transfer across scenes (TranSplat), and scene editing pipelines [2503.22676, 2312.03704, 2410.11419].

Limitations include low-frequency-truncation inherent to SH-only methods (blurry shadows, loss of sharp features), challenges for thin/transparent/refractive materials, out-of-distribution deterioration for hair/accessories or unusual geometry, and, for some pipelines, the need for special capture (e.g., OLAT) or per-scene optimization [2312.03704, 2509.22112, 2410.11419].

## 7. Comparative Table of Major Relightable 3DGS Methods

| Method                          | Geometry           | Appearance Model         | Lighting/BRDF               | Core Relighting Pipeline         |
|----------------------------------|--------------------|-------------------------|-----------------------------|----------------------------------|
| Relightable Gaussian Codec Avatars [2312.03704] | 3D Gaussians (deformable) | SH diffuse, SG specular, learned transfer | SH, SG, latent CVAE, explicit eye model | Front-to-back compositing with radiance transfer |
| Phys3DGS [2409.10335]           | Hybrid mesh+GS     | PBR BRDF + indirect/vis SH | Cook-Torrance/Disney BRDF   | Deferred rendering: G-buffer + screen shading |
| GS³ [2410.11419]                | 3D Gaussians       | Lambertian + mixture of angular Gaussians | Custom reflectance, MLP refinement | Triple splat: shading, shadow, GI  |
| PRTGS [2408.03538]              | 3D Gaussians       | Precomputed SH transfer  | PRT kernel (SH), diffuse+glossy | SH tensor contraction per splat   |
| MGM [2509.22112]                | 2DGS               | PBR (a,ρ,m), no baked light| Cook-Torrance              | Splat attributes + full BRDF integral  |
| BiGS [2408.13370]               | 3D Gaussians       | Bidirectional SH, no normals| Full bidir SH scattering    | SH composition, fully volumetric  |
| PR-ENDO [2411.12510]            | 3D Gaussians       | Diffuse MLP, Cook-Torrance| Endoscopy PBR, hybrid      | Alpha splat + per-splat MLP      |
| TranSplat [2503.22676]          | 2DGS/3DGS          | Spherical Harmonics       | No explicit materials      | SH coefficient remapping per splat|
| RelightAnyone [2601.03357]      | 3DGS (head)        | SH transfer diffuse, SG specular | Two-stage, latent UNets   | Cross-subject mapping, alpha compositing |

All methods combine explicit Gaussian-based geometry with learnable physically based appearance and radiance transfer, harnessing efficient splatting or deferred shading for real-time and interactive relighting. The technical design—choice of SH/SG model, deferred/ray-traced rendering, explicit SDF or hybrid mesh incorporation, and neural extensions—adapts to the required fidelity, performance, and domain context.

Source: https://www.emergentmind.com/topics/relightable-3d-gaussian-splatting-representation