Gaussian in the Dark: Real-Time View Synthesis From Inconsistent Dark Images Using Gaussian Splatting (2408.09130v2)

Abstract: 3D Gaussian Splatting has recently emerged as a powerful representation that can synthesize remarkable novel views using consistent multi-view images as input. However, we notice that images captured in dark environments where the scenes are not fully illuminated can exhibit considerable brightness variations and multi-view inconsistency, which poses great challenges to 3D Gaussian Splatting and severely degrades its performance. To tackle this problem, we propose Gaussian-DK. Observing that inconsistencies are mainly caused by camera imaging, we represent a consistent radiance field of the physical world using a set of anisotropic 3D Gaussians, and design a camera response module to compensate for multi-view inconsistencies. We also introduce a step-based gradient scaling strategy to constrain Gaussians near the camera, which turn out to be floaters, from splitting and cloning. Experiments on our proposed benchmark dataset demonstrate that Gaussian-DK produces high-quality renderings without ghosting and floater artifacts and significantly outperforms existing methods. Furthermore, we can also synthesize light-up images by controlling exposure levels that clearly show details in shadow areas.

• The paper introduces Gaussian-DK, integrating a camera response model into 3D Gaussian Splatting to address exposure inconsistencies in low-light environments.
• The paper achieves superior performance over NeRF-W and HDR-NeRF by using a step-based gradient scaling strategy and a tone mapping CNN to reduce artifacts.
• The paper’s findings pave the way for advanced low-light view synthesis applications such as nighttime navigation and enhanced photography.

Analysis of "Gaussian in the Dark: Real-Time View Synthesis From Inconsistent Dark Images Using Gaussian Splatting"

This paper addresses a persistent challenge in the field of computer vision and graphics: synthesizing consistent and high-quality novel views from images captured in dark environments where traditional multi-view consistency assumptions often fail due to considerable brightness variations and inconsistencies introduced by camera limitations. To overcome this challenge, the authors propose Gaussian-DK, an enhancement of the 3D Gaussian Splatting (3DGS) framework, which is adept at dealing with the inherent inconsistencies found in dark settings.

The primary contribution of Gaussian-DK lies in its novel approach to handling inconsistencies by simulating the exposure variances seen across different camera settings in dim environments. It achieves this by integrating a comprehensive camera response model into the 3DGS framework. This model incorporates exposure levels derived from exposure time, ISO gain, and aperture settings to accurately represent and modulate the radiance field using anisotropic 3D Gaussians. Additionally, it employs a convolutional neural network (CNN) to perform tone mapping, converting modulated radiance values into pixel values that reflect accurate brightness levels while compensating for inter-view inconsistencies.

Notably, Gaussian-DK introduces a step-based gradient scaling strategy aimed at mitigating artifacts known as "floaters" that are prominent in view synthesis tasks involving complex brightness conditions. By refining the optimization process and reducing the tendency of Gaussians to split near the camera, Gaussian-DK demonstrates superior performance in preserving high-frequency details without the typical ghosting artifacts.

Experimental results on a novel dataset, specifically captured in complex real-world dark environments, highlight Gaussian-DK’s capability to outperform baseline methods such as NeRF-W and HDR-NeRF. It achieves significant improvements in PSNR, SSIM, and LPIPS, ensuring both perceptual and quantitative gains while maintaining real-time rendering speeds. The dataset itself, comprising 12 diverse scenes with significant view disparities, sets a new benchmark for evaluating novel view synthesis in low-light conditions.

The implications of these advancements are extensive. Practically, the ability to generate consistent renderings from inconsistent inputs is crucial for applications such as nighttime autonomous navigation and low-light photography enhancement. Theoretically, the work propels understanding of how exposure dynamics can be modeled and compensated within radiance fields, paving the way for further refinements in neural rendering techniques, especially in scenarios with challenging lighting conditions.

Future research could delve into optimizing the computational efficiency of Gaussian-DK, integrating adaptive learning methods for exposure settings, and expanding its application scope to include moving light sources or dynamic environments. Moreover, the discussed limitations regarding exposure level deviations suggest room for enhancing model robustness under extreme exposure variations. Exploration of these avenues could contribute to more adaptive and less resource-intensive view synthesis solutions across diverse environmental conditions.