Recurrent Gaussian Splatting

• RecGS is an iterative technique that refines 3D scene representations by combining Gaussian Splatting with recurrent feedback loops to remove illumination artifacts.
• It uses a recurrent optimization process alternating between rendering and low-frequency caustic extraction via FFT filtering to minimize photometric residuals.
• The approach is applicable to underwater imaging, dynamic scene modeling, and VR/AR, enhancing reconstruction quality under varying lighting conditions.

Recurrent Gaussian Splatting (RecGS) refers to a class of iterative or feedback-driven approaches for refining 3D scene representations using Gaussian primitives, leveraging both spatial and photometric residuals to enhance reconstruction quality over successive iterations. Originating from applications in underwater imaging, RecGS builds upon the explicit, differentiable 3D Gaussian Splatting (3DGS) framework but introduces a recurrent cycle wherein low-frequency illumination inconsistencies are progressively removed, yielding photorealistic, artifact-free reconstructions without reliance on supervised annotated data. The methodology has broader implications for inverse rendering, dynamic scene modeling, and online streaming updates, positioning RecGS as a foundational technique for domains requiring robust, adaptive scene representations.

1. Core Principles and Fundamental Frameworks

RecGS exploits the structure-aware properties of 3DGS to address cases where illumination varies inconsistently across a scene, such as water caustics in seafloor imagery. Unlike standard 2D filtering or supervised deep learning approaches, which operate in the image domain and often struggle to generalize to scenes with non-planar geometry or significant motion, RecGS instantiates a recurrent optimization loop:

• A 3DGS model $G$ is initialized using Structure-from-Motion (SfM) outputs (e.g., COLMAP poses plus keypoints).
• The rendered image $\hat{I}$ is compared to the captured image $I$, computing a residual $R = I - \hat{I}$.
• A low-pass filter (2D FFT retaining the lowest $k$ frequencies, typically $k = 9$) is applied to $R$ to estimate the caustic/illumination component $C$.
• The 3DGS model is re-optimized for the caustic-compensated image $I - C$, minimizing $dist(I - C, \hat{I})$.

This process is iterated, alternating between caustic estimation and 3DGS re-optimization, until convergence is achieved—quantified by the norm $\|C - C_{\text{prev}}\|_2$ falling below a preset threshold (Zhang et al., 14 Jul 2024).

2. Detailed Methodology and Algorithmic Steps

The methodology of RecGS is operationalized with the following workflow:

• Initialization: SfM yields camera parameters and a sparse 3D keypoint cloud, seeds 3DGS Gaussians.
• Rendering Equation: The view synthesis leverages the compositing formula

$$\hat{I} = \sum_i \left( c_i \, \alpha_i \prod_{j < i} (1 - \alpha_j) \right)$$

where $c_i$ is the radiance/color and $\alpha_i$ the opacity of the $i$-th Gaussian.

• Residual and Low-Frequency Estimation: After each rendering cycle, the difference $R$ is filtered with $\text{FFT}/\text{IFFT}$:

$C = \text{ifft}\big( \text{fft}(R)_{[0:k]} \big)$

yielding the additive illumination inconsistency.

• Recurrent Optimization: The process alternates:
  1. Compute $R$,
  2. Extract $C$,
  3. Optimize $G$ for $I - C$,
  4. Check convergence.

The design enables the explicit separation of caustics from true scene structure, progressively improving visual fidelity and consistency in the reconstructed model.

3. Comparative Evaluation and Effectiveness

Experimental comparisons reveal significant distinctions between RecGS and alternative approaches (Zhang et al., 14 Jul 2024):

Method	Generalization	Consistency	Typical Artifact
Joint Optimization (3DGS+Caustic)	Poor	Good	"Consistently wrong" illumination
2D Filtering	Poor	Poor	Artifacts in non-planar geometry
Pretrained Deep Learning	Poor	Variable	Poor transfer to 3D/camera motion
RecGS (Recurrent)	Good	Good	Uniform, correct illumination

Joint optimization strategies were observed to cause over-exposed regions throughout the scene without correcting underlying illumination irregularities. In contrast, RecGS consistently removed caustics, providing stable corrections even under changing camera viewpoints and 3D scene complexity.

4. Extensions, Applications, and Generalization

RecGS, though demonstrated for marine seafloor imaging, is applicable to other domains characterized by illumination inconsistency. Candidate applications include:

• Scenes with spatially non-uniform lighting (e.g., architectural/urban environments with shadows).
• Restoration tasks in aerial imaging, cultural heritage digitization, and dynamic VR/AR contexts.
• Any 3D scene reconstruction problem where supervised data is impractical, making unsupervised, frequency-based methods valuable.

By expanding the frequency-filtering scheme (i.e., tuning $k$), RecGS can be adapted for problems involving non-uniform exposure, brightness compensation, or color drift in reconstructed models.

5. Limitations and Directions for Future Work

Key limitations and prospective improvements identified are:

• The ability of 3DGS models to generalize to novel viewpoints remains constrained by input coverage; reconstructions from sparse or limited views may yield artifacts or depleted fidelity when viewing angles diverge from training data.
• Manual parameter selection, such as the threshold $k$ in the FFT filter, could be replaced with adaptive schemes informed by image statistics or uncertainty quantification.
• Integrating advanced view synthesis strategies or regularization (drawing on ideas from geometric regularization (Kim et al., 16 Jun 2025), adaptive sampling (Kim et al., 15 Jun 2025), or dynamic model compression (Huang et al., 22 Jul 2025)) may enhance robustness and scalability.
• Broader extensions could target dynamic scene modeling, inverse rendering, and streaming pipelines requiring recurrent updates over time.

6. Broader Significance and Relation to Contemporary Research

RecGS signals a shift toward recurrent, unsupervised scene refinement, employing spectral decomposition as an iterative corrective mechanism. The approach connects with recent advances in:

• Probabilistic and adaptive sampling frameworks for densification (Kim et al., 15 Jun 2025).
• Multiview geometric regularization for improved radiance field accuracy (Kim et al., 16 Jun 2025).
• Online and streaming models supporting temporal recurrence and memory-efficient update (Huang et al., 22 Jul 2025).

A notable implication is that structure-aware, recurrent Gaussian Splatting techniques may underpin future methodologies for robust, artifact-free, and scalable 3D reconstruction under challenging illumination, motion, or dynamic conditions.

7. Summary

Recurrent Gaussian Splatting, and specifically the RecGS framework, introduces a principled, iterative approach for removing low-frequency illumination artifacts within explicit 3D Gaussian models. Its reliance on frequency-based residual decomposition and unsupervised recurrent optimization yields significant improvements in rendering consistency and generalization, positioning RecGS as a foundational methodology for a wide range of future applications including underwater imaging, dynamic reconstruction, and any context where illumination inconsistency poses a significant challenge to high-fidelity 3D radiance field modeling (Zhang et al., 14 Jul 2024).