WeatherGS: 3D Scene Reconstruction in Adverse Weather Conditions via Gaussian Splatting (2412.18862v3)

Abstract: 3D Gaussian Splatting (3DGS) has gained significant attention for 3D scene reconstruction, but still suffers from complex outdoor environments, especially under adverse weather. This is because 3DGS treats the artifacts caused by adverse weather as part of the scene and will directly reconstruct them, largely reducing the clarity of the reconstructed scene. To address this challenge, we propose WeatherGS, a 3DGS-based framework for reconstructing clear scenes from multi-view images under different weather conditions. Specifically, we explicitly categorize the multi-weather artifacts into the dense particles and lens occlusions that have very different characters, in which the former are caused by snowflakes and raindrops in the air, and the latter are raised by the precipitation on the camera lens. In light of this, we propose a dense-to-sparse preprocess strategy, which sequentially removes the dense particles by an Atmospheric Effect Filter (AEF) and then extracts the relatively sparse occlusion masks with a Lens Effect Detector (LED). Finally, we train a set of 3D Gaussians by the processed images and generated masks for excluding occluded areas, and accurately recover the underlying clear scene by Gaussian splatting. We conduct a diverse and challenging benchmark to facilitate the evaluation of 3D reconstruction under complex weather scenarios. Extensive experiments on this benchmark demonstrate that our WeatherGS consistently produces high-quality, clean scenes across various weather scenarios, outperforming existing state-of-the-art methods. See project page:https://jumponthemoon.github.io/weather-gs.

• The paper introduces WeatherGS, a 3D Gaussian Splatting framework with preprocessing steps (AEF, LED) to effectively reconstruct scenes despite rain and snow artifacts.
• WeatherGS employs an Atmospheric Effect Filter and Lens Effect Detector to clean multi-view images before reconstruction with a 3D Gaussian Splatting method robust to small-scale inconsistencies.
• Validated on synthetic and real-world datasets, WeatherGS demonstrates superior performance (PSNR, SSIM, LPIPS) compared to existing methods for 3D reconstruction in rainy and snowy conditions, benefiting applications like autonomous driving.

An Analysis of WeatherGS: 3D Scene Reconstruction in Adverse Weather Conditions via Gaussian Splatting

The paper "WeatherGS: 3D Scene Reconstruction in Adverse Weather Conditions via Gaussian Splatting" introduces a method to address the challenges posed by adverse weather on 3D scene reconstruction. The primary innovation of WeatherGS is its ability to mitigate the effects of dense particles and lens occlusions caused by rain and snow, using a 3D Gaussian Splatting (3DGS) framework. This approach significantly enhances the quality of reconstructed scenes in challenging environmental conditions, a context that has not been extensively explored in the field of 3D reconstruction.

Methodology

WeatherGS operates by integrating a dense-to-sparse preprocessing strategy. This method tackles the complexity of weather artifacts by dealing with dense weather particles and occlusions as distinct phenomena, each with unique properties that require tailored handling. Initially, the Atmospheric Effect Filter (AEF) removes dense particles such as raindrops or snowflakes. Leveraging diffusion models enriched with weather-specific priors allows this filter to effectively clear small-scale noise, resulting in clearer images that preserve scene details.

Following AEF, the Lens Effect Detector (LED) identifies and masks occlusions on the camera lens. These two preprocessing steps generate clean multi-view images, effectively excluding irrelevant data that could compromise scene reconstruction quality.

The paper highlights the adaptability of 3DGS, which combines visual fidelity with computational efficiency. Despite traditional methods from the Neural Radiance Field (NeRF) family facing difficulties with small-scale inconsistencies in dynamic and adverse conditions, 3DGS uses an explicit radiance field and Gaussian distribution to filter and smooth weather-related artifacts, maintaining scene clarity.

Experimentation

WeatherGS is validated through extensive experimentation on a newly developed benchmark comprising both synthetic and real-world datasets. The authors used scenarios depicting snowy and rainy weather conditions to test the efficacy of their method across varied and complex environments. The contrast with methods such as DerainNeRF and vanilla implementations of 3DGS and NeRF indicates WeatherGS’s superior ability to render high-quality scenes by effectively eliminating weather-induced disturbances.

Quantitative measures using PSNR, SSIM, and LPIPS strongly favor WeatherGS, showcasing its efficiency in maintaining structural similarity with high perceptual quality relative to clean scenes. This effectiveness is evident despite potential slight distortions introduced during preprocessing by diffusion models and task-specific plugins.

Implications and Future Developments

The implications of this research are substantial for fields reliant on precise environmental automations, such as autonomous driving, realistic virtual reality renderings, and robotics operating in adverse weather conditions. The ability to construct clear 3D scenes in poor visibility situates WeatherGS as a crucial component for applications requiring high fidelity and real-time operational capability.

The paper suggests several future research avenues, such as enhancing preprocessing stages to reduce potential distortions further and expanding the applicability of WeatherGS to more diverse and especially unpredictable weather conditions. Additionally, integrating machine learning models to predict and adapt to environmental conditions could further optimize the preprocessing efficiency.

In conclusion, the WeatherGS framework significantly advances the field of 3D scene reconstruction under adverse weather conditions. By integrating cutting-edge preprocessing techniques with the robustness of 3D Gaussian Splatting, this method effectively addresses the long-standing challenge of maintaining scene integrity amidst environmental noise. As the research community continues to tackle the challenges posed by environmental variability, WeatherGS serves as a blueprint for future developments in this pivotal domain.