SeaSplat: Representing Underwater Scenes with 3D Gaussian Splatting and a Physically Grounded Image Formation Model

Abstract: We introduce SeaSplat, a method to enable real-time rendering of underwater scenes leveraging recent advances in 3D radiance fields. Underwater scenes are challenging visual environments, as rendering through a medium such as water introduces both range and color dependent effects on image capture. We constrain 3D Gaussian Splatting (3DGS), a recent advance in radiance fields enabling rapid training and real-time rendering of full 3D scenes, with a physically grounded underwater image formation model. Applying SeaSplat to the real-world scenes from SeaThru-NeRF dataset, a scene collected by an underwater vehicle in the US Virgin Islands, and simulation-degraded real-world scenes, not only do we see increased quantitative performance on rendering novel viewpoints from the scene with the medium present, but are also able to recover the underlying true color of the scene and restore renders to be without the presence of the intervening medium. We show that the underwater image formation helps learn scene structure, with better depth maps, as well as show that our improvements maintain the significant computational improvements afforded by leveraging a 3D Gaussian representation.

• The paper presents a novel method that combines 3D Gaussian Splatting with a physical underwater image model to address attenuation and backscatter challenges.
• It integrates real-time, efficient 3D rendering with physics-based corrections to achieve accurate scene reconstruction and depth estimation.
• Experimental results show enhanced image fidelity and coherent depth maps, offering practical benefits for underwater navigation and exploration.

SeaSplat: Representing Underwater Scenes with 3D Gaussian Splatting and a Physically Grounded Image Formation Model

This paper introduces SeaSplat, a novel method for real-time rendering of underwater scenes through the integration of 3D Gaussian Splatting (3DGS) and an underwater image formation model. This approach aims to address the unique challenges presented by underwater visual environments, such as range-dependent and color-dependent image degradation due to attenuation and backscatter effects.

Introduction

Underwater imaging presents distinct challenges due to the medium through which images are captured. Water introduces attenuation, which is wavelength-dependent, causing certain colors (specifically red) to diminish more quickly with distance, and backscatter, leading to a hazy appearance. Traditional computer vision methods are often ineffective in such conditions. Therefore, integrating a physically grounded image formation model with advances in 3D radiance fields, particularly 3DGS, provides a potent solution for accurate and efficient representation of underwater scenes.

Methodology

SeaSplat innovatively combines 3DGS with a model of underwater image formation. The method leverages 3DGS for efficient, real-time photorealistic rendering of 3D environments, while incorporating a physics-based underwater model to simulate the effects of attenuation and backscatter. This combination allows SeaSplat to simultaneously restore the true color of the scene and accurately estimate the scene’s geometry.

Key components of SeaSplat:

• 3D Gaussian Representation: The scene is parameterized using 3D Gaussian distributions, allowing for efficient rendering and accurate scene representation.
• Underwater Image Formation Model: The model considers wavelength-specific attenuation and backscatter effects, augmenting traditional 3D radiance fields to account for underwater conditions.
• Optimization Process: A multi-faceted loss function is used, incorporating standard reconstruction losses from 3DGS, as well as additional terms to handle backscatter, color consistency, and depth smoothing.

Through this approach, SeaSplat not only enhances the quality of novel view synthesis but also provides more coherent and physically plausible depth maps compared to standard 3DGS.

Results

Quantitative Analysis

SeaSplat demonstrates superior performance in novel view rendering compared to both SeaThru-NeRF and vanilla 3D Gaussian Splatting across a range of datasets, including:

• SeaThru-NeRF datasets (Curaçao, Japanese Gardens, Panama, IUI3)
• Synthetic scenes (simulated fog and underwater conditions)
• Real-world data from underwater vehicle surveys (Salt Pond)

Performance metrics include PSNR, SSIM, and LPIPS, with SeaSplat consistently showing higher fidelity and coherence in rendered images and estimated depth maps.

Computational Efficiency

The computational efficiency of SeaSplat is worth noting. It maintains the real-time rendering capabilities of 3DGS, showing minimal additional computational overhead. This efficiency is critical for potential real-world applications where computational resources might be constrained.

Discussion

Implications:

SeaSplat’s integration of a physics-based model for underwater environments offers significant practical and theoretical advancements. For practical applications, this method can be pivotal in improving underwater navigation, collision avoidance, and the autonomous exploration of underwater environments by enhancing the visual clarity and accuracy of scene representations. Theoretically, it provides a framework for future research in adapting radiance fields to various non-atmospheric media.

Future Developments:

Key areas for future exploration include adapting SeaSplat for dynamic underwater scenes and modeling more complex interactions between light and underwater surfaces. Enhancing the system to function in real-time adaptive contexts, such as on autonomous underwater vehicles, could also expand its utility and impact.

Conclusion

SeaSplat provides a robust and efficient method for rendering underwater scenes by augmenting 3D Gaussian Splatting with a physically grounded image formation model. It achieves high fidelity in novel view synthesis and accurate depth estimation, while maintaining the computational efficiency necessary for real-time applications. The advancements presented in this paper mark a significant step forward in the representation and understanding of underwater visual environments, laying the groundwork for future innovations in this challenging and dynamic field.