R3GW: Relightable 3D Gaussians for Outdoor Scenes

• R3GW is a method that uses enhanced 3D Gaussian splatting to enable physically-based relighting and photorealistic reconstruction of outdoor scenes.
• It partitions scene elements into relightable foreground and non-reflective sky Gaussians, leveraging spherical harmonics and PBR for accurate illumination modeling.
• Experimental evaluations on the NeRF-OSR dataset demonstrate competitive PSNR and SSIM metrics, confirming its effectiveness for real-world outdoor scene reconstruction.

R3GW (Relightable 3D Gaussians for Outdoor Scenes in the Wild) is a method designed to enable physically-based relighting and photorealistic novel-view synthesis of outdoor scenes captured under varying illumination, using an enhanced 3D Gaussian Splatting (3DGS) representation. R3GW addresses the limitations of standard 3DGS in modeling scene illumination and reconstructing scenes from uncontrolled datasets, such as Internet photo collections, by incorporating explicit relightable foreground modeling and a differentiated non-reflective background (sky). The method leverages physically-based rendering (PBR) models, spherical harmonics (SH) environment illumination, and a dedicated sky representation to achieve state-of-the-art quantitative and qualitative performance on the NeRF-OSR dataset (Corona et al., 3 Mar 2026).

1. Scene Representation and Gaussian Parameterization

R3GW extends 3D Gaussian Splatting by partitioning the scene elements into two non-overlapping sets of Gaussians: relightable foreground Gaussians and non-reflective sky Gaussians. Each Gaussian, indexed by $i$, is characterized by:

• Position: $p_i \in \mathbb{R}^3$
• Covariance: $\Sigma_i = R_i S_i S_i^T R_i^T$, with $R_i \in SO(3)$ and $S_i = \operatorname{diag}(s_x, s_y, s_z)$
• Opacity: $o_i \in [0, 1]$
• Foreground Gaussians encode: albedo $a_i \in [0, 1]^3$, roughness $\rho_i \in [0, 1]$, and normal $n_i$
• Sky Gaussians encode only a color $c_i$, computed from per-image sky SH coefficients

Rendering employs a “splatting” procedure and alpha compositing:

$p_i \in \mathbb{R}^3$0

A boolean tag distinguishes foreground and sky Gaussians, allowing role-specific loss enforcement.

2. Physically-Based Rendering and Illumination Modeling

R3GW integrates a physically-based rendering approach for the foreground using the rendering equation for non-emissive surfaces:

$p_i \in \mathbb{R}^3$1

where $p_i \in \mathbb{R}^3$2 is a microfacet Cook–Torrance BRDF parameterized in the Disney style (fixing metallic = 0), splitting into:

• Diffuse (Lambertian): $p_i \in \mathbb{R}^3$3
• Specular (microfacet): $p_i \in \mathbb{R}^3$4

Environment illumination for relighting is represented by degree-4 SH for the light (foreground) and degree-1 SH for the sky (background). Foreground colors are computed as the sum of a diffuse and specular component:

• $p_i \in \mathbb{R}^3$5
• $p_i \in \mathbb{R}^3$6 is calculated via the split-sum approximation, using blurred SH coefficients whose bandwidth corresponds to roughness$p_i \in \mathbb{R}^3$7

An MLP, conditioned on a per-image latent embedding $p_i \in \mathbb{R}^3$8, predicts the SH lighting and sky coefficients per view. At test time, arbitrary HDR environment maps can be incorporated via SH projection.

3. Learning Scheme and Loss Structure

The R3GW optimization procedure initializes:

• Foreground Gaussians from COLMAP SfM point clouds (positions), with randomized appearance and geometric attributes
• Sky Gaussians by uniform sampling on the upper hemisphere of a reference radius

Trainable parameters include all Gaussian attributes (except flags and color), per-image appearance embeddings, and MLP weights.

The total loss is:

$p_i \in \mathbb{R}^3$9

Component losses include:

• $\Sigma_i = R_i S_i S_i^T R_i^T$0: photometric reconstruction (weighted $\Sigma_i = R_i S_i S_i^T R_i^T$1 and D-SSIM)
• $\Sigma_i = R_i S_i S_i^T R_i^T$2: light-positivity regularization [LumiGauss]
• $\Sigma_i = R_i S_i S_i^T R_i^T$3: normal consistency [2D-GS]
• $\Sigma_i = R_i S_i S_i^T R_i^T$4: penalizes small Gaussian axes for sheet-likeness [NeuSG]
• $\Sigma_i = R_i S_i S_i^T R_i^T$5: enforces foreground/sky separation using ground-truth masks
• $\Sigma_i = R_i S_i S_i^T R_i^T$6: encourages sky Gaussians to be behind foreground

Hyperparameters control the influence of each term ($\Sigma_i = R_i S_i S_i^T R_i^T$7). Occluders and the sky are masked out during loss computation.

4. Dedicated Sky Representation

A distinct set of Gaussians is assigned to model the sky, with positions in spherical coordinates $\Sigma_i = R_i S_i S_i^T R_i^T$8, and color from per-image SH:

$\Sigma_i = R_i S_i S_i^T R_i^T$9

where $R_i \in SO(3)$0 are first-order SH basis functions. Losses ($R_i \in SO(3)$1, $R_i \in SO(3)$2) ensure sky Gaussians are spatially restricted and reside behind all foreground elements. This architecture mitigates depth-leak and halo artifacts at the skyline, leading to improved foreground-background boundaries.

5. Experimental Evaluation and Quantitative Results

R3GW is evaluated on the NeRF-OSR dataset, consisting of three outdoor scenes (lk2, lwp, st) with unconstrained multi-view photos and ground-truth LDR environment maps.

Key evaluation metrics (masked to foreground):

• PSNR (↑)
• SSIM (↑)
• MSE (↓)
• MAE (↓)

Summary of results (averaged over three scenes):

Method	PSNR	SSIM	MSE	MAE
NeRF-OSR	16.56	0.509	0.024	0.111
SR-TensoRF	16.56	0.609	0.025	0.100
LumiGauss*	18.21	0.736	0.017	0.096
R3GW	18.11	0.739	0.018	0.098

R3GW matches or exceeds prior Gaussian-based approaches in SSIM and PSNR on select scenes, and outperforms unshadowed LumiGauss overall. The method supports photorealistic relighting under arbitrary HDR illumination, including plausible specular highlights and global color shifts (Corona et al., 3 Mar 2026).

6. Implementation Details, Limitations, and Future Directions

Implementation is based on Kerbl et al.’s 3DGS CUDA code. The architecture uses three 256-unit MLP layers for the appearance embedding, with specialized heads for sky and light SH. Training utilizes the Adam optimizer with task-specific learning rates and warm-up stages for stability. The method inherits adaptive Gaussian densification and pruning techniques from 3DGS.

Known limitations:

• No explicit cast shadow or directional sunlight modeling—shadows and indirect effects are only approximated by SH basis truncation, leading to soft speculars and weak shadow detail under clear skies.
• Extensions may include explicit sun-light modeling with shadow mapping in the 3DGS architecture or tighter coupling between sky and foreground illumination.

R3GW provides a fully detailed pipeline for relightable 3D Gaussian Splatting of outdoor scenes in unconstrained illumination, combining robust scene segmentation, nonlinear physically-based lighting models, and scalable optimization (Corona et al., 3 Mar 2026).