ROSGS: Relightable Outdoor Scenes with Gaussian Splatting

• ROSGS is a method for relightable outdoor scene synthesis that decouples geometry, texture, and illumination to achieve high-quality, real-time rendering.
• It utilizes 2D Gaussian Splatting enhanced by monocular normal priors and learned appearance embeddings to precisely reconstruct scene geometry.
• The hybrid lighting model combines a spherical Gaussian for sharp sunlight effects with spherical harmonics for ambient skylight, ensuring accurate and efficient illumination.

ROSGS is the abbreviation for "Relightable Outdoor Scenes with Gaussian Splatting," a method developed for efficient, high-quality relighting of unbounded outdoor environments. It addresses critical limitations in conventional neural rendering—particularly the computational inefficiency of neural radiance field approaches and the inability of existing Gaussian splatting methods to model high-frequency lighting effects typical of outdoor sunlight. ROSGS implements a two-stage pipeline that decomposes outdoor scene appearance into geometry, texture, and illumination, combining a compact 2D Gaussian Splatting geometry initialization with a hybrid, physically-motivated lighting model capable of representing both sharp, directional sunlight and low-frequency diffuse skylight.

1. Problem Scope and Motivation

Outdoor image-based rendering involves significant complexity due to unconstrained illumination, weather-induced variability, and the unbounded nature of most scenes. Many traditional neural rendering methods, such as NeRF and its lighting-aware variants, are limited by their heavy computational footprint and low-frequency lighting models that cannot produce precise cast shadows and high-frequency lighting variations. Recent 3D Gaussian Splatting (3DGS) techniques provide efficient rendering, but their relighting capabilities are hindered by similar issues in lighting representation. ROSGS is explicitly designed to overcome these challenges: the method achieves real-time rendering and superior relighting by decoupling geometry, texture, and complex illumination components, producing outputs that maintain physically plausible lighting responses under new outdoor scenarios.

2. Two-Stage Technical Pipeline

ROSGS executes relightable scene synthesis via a two-stage process:

• Stage One: Geometry Reconstruction using 2DGS
  • The initial phase employs 2D Gaussian Splatting (2DGS) to recover scene geometry from multi-view image collections. Gaussian splats are parameterized and placed such that their combined projections accurately reconstruct scene structure with high coverage and efficiency.
  • An appearance transformation module introduces learned per-Gaussian and per-image embeddings to harmonize appearance across varying illumination and observation angles.
  • Monocular normal priors—such as those derived from single-view surface normal estimation—regularize the optimization, ensuring stable surface orientation reconstruction even with weak texture or strong shadowing.
  • Output depths are fused to form a watertight mesh supporting downstream shadow ray tracing and accurate direct illumination computation.
• Stage Two: Texture and Lighting Decomposition with Hybrid Outdoor Lighting Model
  • Fixing geometry from Stage One, the next phase separates reflectance (albedo) and illumination.
  • The hybrid lighting model explicitly splits outdoor lighting into:
  • Sunlight, modeled as a spherical Gaussian (SG): a parametric, high-frequency distribution capturing the sun’s direction and intensity, as in

  $$SG_{(\mu, \lambda, \xi)}(\nu) = \mu \cdot \exp(\lambda (\nu \cdot \xi - 1))$$

  where $\mu$ controls amplitude (intensity/color), $\lambda$ controls the sharpness, $\xi$ gives sun direction, and $\nu$ is the query direction. - Skylight, modeled via Precomputed Radiance Transfer (PRT) and Spherical Harmonics (SH): enables efficient, low-frequency description of ambient, directionally diffuse illumination:

  $$I_{\text{amb}} = \mathcal{S}\left( \sum_{j=1}^{(n+1)^2} l_c^j \cdot t_k^j \right)$$

  where $l_c$ are global SH coefficients, $t_k$ are per-Gaussian radiance transfer SH coefficients, and $\mathcal{S}(\cdot)$ is the 2DGS-based rasterization. - This design allows both high-efficiency rendering and accurate simulation of realistic sun shading effects and ambient sky gradients.

3. Mathematical Formulation

ROSGS leverages a compact, analytically tractable lighting decomposition:

• Sunlight is captured by a single-view-aligned Spherical Gaussian, allowing direct evaluation of directional irradiance for each scene surface element.
• Sky radiance is encoded with $n^\text{th}$-order Spherical Harmonics (empirically, $n=2$), providing a balance between efficiency and completeness for low-frequency lighting.
• Texture-albedo separation is enforced by fixing geometry and optimizing both per-pixel albedo and lighting to minimize the error between observed and rendered images across all training views.

The overall process adheres to the physically-based rendering equation, but with analytic integration for both direct (sun) and indirect (sky) components, thus affording rapid evaluation and the ability to adjust lighting for relighting applications.

4. Performance Evaluation

ROSGS demonstrates marked improvements over prior methods in both relighting quality and computational efficiency:

• Quantitative Metrics: Across several outdoor datasets, ROSGS achieves superior Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and lower Mean Squared Error (MSE) in comparison to baseline methods such as NeRF-OSR, SOL-NeRF, ReCap, and LumiGauss. Normal estimation quality, as measured by mean absolute error (degrees), surpasses prior art by leveraging monocular priors during geometry optimization.
• Rendering Efficiency: The 2DGS-based pipeline operates at real-time frame rates, outpacing typical NeRF-based relighting systems which suffer from neural evaluation bottlenecks. Training times are reduced via efficient decoupling of geometry and lighting and avoidance of per-pixel neural network evaluations.
• Qualitative Analysis: Visualizations illustrate sharper, more realistic cast shadows and improved texture-illumination separation compared to competing methods, without the characteristic blurring or loss of shadow fidelity from low-frequency only approaches.

5. Applications and Use Cases

ROSGS has direct relevance in several high-value domains requiring robust and efficient outdoor relighting:

• Augmented Reality (AR): Real-time, relightable outdoor environments enable seamless localization and rendering of virtual objects with photorealistic shadowing and ambient response.
• Urban Planning, Virtual Production: Architectural visualization benefits from manipulating sunlight and environmental conditions interactively in large-scale, unbounded scenes.
• Autonomous Systems: Relightable, metric-accurate scene reconstructions support the simulation and validation of computer vision systems under realistic, time-varying illumination.

A plausible implication is that by decoupling geometry and using hybrid physically-based lighting models in Gaussian Splatting frameworks, future scene reconstruction systems can approach both the generality and efficiency required for online, dynamic relighting scenarios in large environments.

6. Future Directions and Limitations

Opportunities for further development include:

• Extension to more complex, non-Lambertian materials (reflective, glossy, etc.) within the same two-stage pipeline paradigm.
• Incorporating dynamic elements or transient outdoor phenomena (e.g. moving clouds, precipitation), possibly through additional hybrid neural–analytic models.
• Further reductions in memory and computation for deployments on edge devices through optimized splat parameterization and advanced SH compression schemes.

The empirical results and design choices in ROSGS suggest a broader trend toward hybrid representations—combining analytic lighting, efficient geometry splatting, and learned appearance—that can fundamentally advance the field of outdoor scene relighting.

7. Summary Table: ROSGS Pipeline Features

Component	Methodology	Primary Advantage
Geometry Reconstruction	2D Gaussian Splatting + Monocular normal priors	Compactness, fidelity, real-time operation
Direct Lighting Model	Spherical Gaussian (SG)	Captures sharp, directional sunlight effects
Skylight/Ambient Model	Spherical Harmonics (SH) via PRT	Efficient, robust ambient simulation
Texture/Lighting Decoupling	Fixed-geometry optimization	Improved separation, accurate relighting
Rendering Performance	Analytic + rasterization	Real-time, scalable

In conclusion, ROSGS establishes a principled and efficient foundation for advanced outdoor scene relighting tasks by leveraging compact Gaussian splatting, physically-based, hybrid lighting models, and rigorous decomposition of geometry, texture, and illumination. Its advances in both rendering accuracy and performance make it a promising solution for future research and emerging applications requiring photorealistic, interactive manipulation of outdoor environments (Liao et al., 14 Sep 2025).