---
title: 'Triple Gaussian Splatting: Real-Time Relighting'
url: https://www.emergentmind.com/topics/triple-gaussian-splatting
type: topic
---

# Triple Gaussian Splatting: Real-Time Relighting

Triple Gaussian Splatting (GS³; pronounced "GS cubed") is a computational framework for real-time, physically-based relighting and novel-view synthesis of objects from multi-view One-Light-At-a-Time (OLAT) image sets. GS³ represents a scene via a cloud of anisotropic, spatially-situated Gaussians, each equipped with a learned, view- and illumination-dependent reflectance model combining Lambertian and a mixture of angular Gaussians (“angular splatting”). The method unifies geometry estimation, direct and indirect reflectance, shadow computation, and global illumination effects within a fully differentiable, deferred-shading pipeline. GS³ achieves order-of-magnitude speedup over neural inverse rendering approaches, while rendering complex view- and light-dependent effects such as anisotropic specularities, translucency, and self-shadow at 90 frames per second on commodity GPUs [2410.11419].

## 1. Problem Formulation and Motivation

GS³ targets the task of generating photorealistic images of an object under arbitrary views and point-light configurations. Given 500–2,000 multi-view OLAT photographs with known camera and point-light poses, the problem is to learn a scene representation supporting real-time (≈90 fps) photorealistic relighting, including direct illumination, view-dependent effects, self-shadowing, and soft indirect light.

Prior representations exhibit critical limitations: “vanilla” 3D Gaussian Splatting encodes static environment lighting (often using spherical harmonics) and fails on novel light directions or strong view/light effects. Mesh or point-based relighting may require costly ray tracing or precomputed visibilities, and are brittle for translucent or anisotropic geometry. Neural fields (NeRF-derivatives) provide high fidelity but with tens-of-hours training and slow inference (<1 fps). GS³ addresses these issues by simultaneously optimizing geometric, reflectance, shadow, and indirect-light parameters end-to-end via a splatting-based renderer, lowering both training (40–70 minutes) and inference budgets [2410.11419].

## 2. Scene Representation and Reflectance Modeling

A GS³ scene comprises $N$ anisotropic 3D “spatial Gaussians.” Each spatial Gaussian $i$ consists of:
- Position $\mu_i \in \mathbb{R}^3$
- Covariance $\Sigma_i = R_i S_i S_i^{\mathsf{T}} R_i^{\mathsf{T}}$ (scaling $S_i \in \mathbb{R}^{3\times3}$ and rotation $R_i \in \mathrm{SO}(3)$)
- Opacity $\gamma_i$
- Learned reflectance $f_i(\omega_o, \omega_i)$, with outgoing direction $\omega_o$ and incident light direction $\omega_i$ in the local shading frame

The spatial density at point $p$ is:
$$
G_{\mathrm{spa}, i}(p) = \exp\left(-\tfrac{1}{2}(p-\mu_i)^{\mathsf{T}} \Sigma_i^{-1} (p-\mu_i)\right)
$$

Reflectance per Gaussian is split into a diffuse and a specular term:
$$
f_i(\omega_o, \omega_i) = \rho_{d,i}\, f_d(n_i \cdot \omega_i) + \rho_{s,i}\, f_s(\omega_o, \omega_i)
$$
where $\rho_{d,i},\ \rho_{s,i} \in \mathbb{R}^3$ are the RGB diffuse and specular albedos, $n_i$ is the learned shading normal.

- **Diffuse**: Modified Lambertian,
$$
f_d(\omega_i) = \frac{\mathrm{ELU}(n \cdot \omega_i) + \varepsilon(1 - 1/e)}{(1 + \varepsilon(1 - 1/e))\pi},\quad \varepsilon=0.01
$$
- **Specular**: Mixture of $M$ shared “angular Gaussians” (anisotropic SGs) applied to the half-vector $h = (\omega_o + \omega_i)/\|\omega_o + \omega_i\|$,
$$
f_s(\omega_o, \omega_i) = \sum_{j=1}^M \alpha_{i,j}\, G_{\mathrm{ang},j}(h)
$$
Each basis angular Gaussian $G_{\mathrm{ang},j}$ is parameterized by an orthonormal frame $[x_j, y_j, z_j]$ and widths $(\sigma_{x,j}, \sigma_{y,j}, \sigma_{z,j})$:
$$
G_{\mathrm{ang}}(h) = \frac{1}{\sigma_z}\, \exp\Bigg[ -\frac{1}{2}\left( \frac{ \arccos(h \cdot z_j)\, \sqrt{ ( (h \cdot x_j)/\sigma_x )^2 + ( (h \cdot y_j)/\sigma_y )^2 } }{ \sigma_z } \right)^2 \Bigg]
$$
Only the $\alpha_{i,j}$ are learned per Gaussian; the basis is shared scene-wide.

## 3. Triple Splatting Pipeline

GS³ employs a deferred shading strategy with three sequential “splatting” passes per frame:

**1. Appearance (Shading) Splatting**
- For each spatial Gaussian, $f_i(\omega_o, \omega_i)$ is projected to the screen as a 2D ellipse, accumulated as:
$$
\zeta(p) = \sum_j c_j \beta_j \gamma_j T_j,\quad T_j = \prod_{k < j}(1 - \beta_k \gamma_k)
$$
with $\beta_j$ the projected spatial density at the pixel.

**2. Shadow Splatting and MLP Refinement**
- Each Gaussian is projected in light space to generate a “shadow map.” Opacities along each shadow ray are accumulated, yielding raw visibilities $T_i$.
- Per-Gaussian visibilities $(T_i, \omega_i, \mu_i, \ell_i)$ (with $\ell_i \in \mathbb{R}^6$ learned latent) are processed by a 3-layer, 32-unit-per-layer MLP $\Phi$ ($S_i = \Phi(T_i, \omega_i; \mu_i, \ell_i)$), outputting refined shadow values $S_i$ (leaky ReLU activations, sigmoid output).
- The resulting $S_i \cdot \rho_{\mathrm{color},i}$ is splatted onto the image grid as the shadow mask.

**3. Global Illumination Compensation MLP**
- Each spatial Gaussian outputs a residual color $R_i = \Psi(\omega_o; \mu_i, \ell_i)$ via a 3-layer, 128-unit MLP $\Psi$ (leaky ReLU + sigmoid).
- All $R_i$ are splatted to produce a global-illumination correction image.

The final image is composed as:
$$
\text{pixel color} = \text{shading} \times \text{shadow} + \text{residual}
$$

## 4. Training Protocols and Implementation

- **Loss function**: Blended $L_1$ and D-SSIM,
$$
L = (1-\lambda)\, \|I_{\mathrm{pred}} - I_{\mathrm{gt}}\|_1 + \lambda\, D\text{-SSIM}(I_{\mathrm{pred}} - I_{\mathrm{gt}})
$$
with $\lambda = 0.2$.
- **Initialization and optimization**:
  - Geometry & opacities as in static GS.
  - Angular Gaussians: $\sigma_z \sim \mathrm{Uniform}(0.13, 0.69),\ \sigma_x = 0.5,\ \sigma_y = 1.0,\ \alpha=0.5$.
  - Two-stage schedule: Stage 1 (15k iters) uses only diffuse reflectance, stabilizing normals; Stage 2 (100k iters) enables full model (specular, shadows, residuals).
  - Adam optimizer ($\beta_1=0.9$), learning rates $1\mathrm{e}{-2}$ with angular Gaussian parameters decaying to $1\mathrm{e}{-4}$ by late training.
- **Datasets**: NeRF-rendered, OpenSVBRDF, learned-scan, handheld-flash photographs, professional lightstage.
- **Compute and resources**: 120k–750k spatial Gaussians, $M=8$ angular bases, 40–70 min training on an RTX 4090. Inference at $\sim$90 fps (512$\times$512), with memory use only modestly above static GS.

## 5. Quantitative and Qualitative Evaluation

A summary of quantitative results across representative methods on standard relighting metrics (averaged over test views and lights):

| Method                                | PSNR (↑) | SSIM (↑) | LPIPS (↓) | Runtime   |
|----------------------------------------|----------|----------|-----------|-----------|
| Ours (GS³)                            | 34.2     | 0.93     | 0.07      | 90 fps    |
| NRHints [Zeng et al. ’23]              | 29.9     | 0.92     | 0.09      | <1 fps    |
| NRTF [Lyu et al. ’22]                  | 30.4     | 0.96     | 0.04      | 0.3 fps   |
| OSF [Yu et al. ’23]                    | 26.1     | 0.94     | 0.05      | ~1 fps    |
| GaussianShader [Jiang ’23]             | 29.3     | 0.94     | 0.06      | 60 fps    |
| GS-IR [Liang ’23]                      | 29.1     | 0.93     | 0.08      | 60 fps    |
| Relightable3DGaussian [Gao ’23]        | 30.2     | 0.95     | 0.05      | 60 fps    |
| TensoIR [Jin ’23]                      | 31.7     | 0.96     | 0.05      | 60 fps    |

Qualitatively, GS³ reproduces intricate relighting phenomena:
- Furballs and subsurface-scattering cups display convincing self-shadow and translucency
- Strong, highly anisotropic highlights on metallic and textile surfaces, attributed to the angular Gaussian mixture
- Accurate self-shadowing in highly occluded scenes (e.g., LEGO assemblies)

## 6. Discussion, Ablations, and Limitations

- *Angular Gaussians ($M$):* $M=4$ suffices for moderate specular lobes; $M\geq8$ is required for modeling sharp glints. Beyond $M=16$, gains diminish.
- *Shadow MLP Width:* Reducing hidden units from 32 to 16 increases shadow noise by $\approx20\%$. Removal of the MLP leads to visible blockiness and aliasing in shadows.
- *Global Illumination MLP:* Disabling causes $\approx15\%$ average increase in residual $L_1$ error; removing direct shading entirely leaves indirect components unmodelled.

**Strengths:**
- Integrates geometry, reflectance, shadowing, and indirect lighting into a single, differentiable, end-to-end optimized pipeline
- Achieves real-time rendering (90 fps) at quality levels on par or superior to NeRF-based relighting, which remains orders of magnitude slower
- Handles challenging cases (translucent, anisotropic, furry) without per-object prior assumptions

**Limitations and Future Directions:**
- Does not support explicit modeling of fully transparent, refractive materials; suggested extension is a differentiable ray-caster replacing the residual MLP
- The fidelity of shadow edges is constrained by the Gaussian cloud’s resolution; further density control or multiscale splatting modes are possible remedies
- Acquisition burden could be reduced via learned illumination multiplexing strategies

GS³’s hybrid approach—combining flexible per-Gaussian reflectance functions, deferred appearance/shadow/global illumination splatting, and learned MLP corrections—enables photorealistic relighting and novel view synthesis at unprecedented interactive speeds, with broad applicability across digitally scanned and real-world captured objects [2410.11419].

Source: https://www.emergentmind.com/topics/triple-gaussian-splatting