Deblurring 3D Gaussian Splatting (2401.00834v3)

Abstract: Recent studies in Radiance Fields have paved the robust way for novel view synthesis with their photorealistic rendering quality. Nevertheless, they usually employ neural networks and volumetric rendering, which are costly to train and impede their broad use in various real-time applications due to the lengthy rendering time. Lately 3D Gaussians splatting-based approach has been proposed to model the 3D scene, and it achieves remarkable visual quality while rendering the images in real-time. However, it suffers from severe degradation in the rendering quality if the training images are blurry. Blurriness commonly occurs due to the lens defocusing, object motion, and camera shake, and it inevitably intervenes in clean image acquisition. Several previous studies have attempted to render clean and sharp images from blurry input images using neural fields. The majority of those works, however, are designed only for volumetric rendering-based neural radiance fields and are not straightforwardly applicable to rasterization-based 3D Gaussian splatting methods. Thus, we propose a novel real-time deblurring framework, Deblurring 3D Gaussian Splatting, using a small Multi-Layer Perceptron (MLP) that manipulates the covariance of each 3D Gaussian to model the scene blurriness. While Deblurring 3D Gaussian Splatting can still enjoy real-time rendering, it can reconstruct fine and sharp details from blurry images. A variety of experiments have been conducted on the benchmark, and the results have revealed the effectiveness of our approach for deblurring. Qualitative results are available at https://benhenryl.github.io/Deblurring-3D-Gaussian-Splatting/

• The paper introduces a deblurring technique that adjusts 3D Gaussian covariances using a small MLP to simulate pixel intermingling for accurate real-time rendering.
• The method improves scene detail by augmenting sparse point clouds with additional valid-color points and employing depth-based pruning.
• Achieving rendering speeds over 200 FPS, the framework delivers state-of-the-art quality while enabling efficient novel view synthesis.

Introduction

The introduction of Neural Radiance Fields (NeRF) revolutionized the approach to novel view synthesis (NVS), providing photorealistic scene reconstructions critical for various domains. However, blurriness in training images—commonly resulting from lens defocusing, motion blur, and camera shake—remains a significant challenge in rendering high-fidelity images. Until recently, volumetric rendering techniques tied to NeRF required heavy computational resources and time, hampering their application for real-time tasks.

Advancements in Speed and Real-Time Rendering

A method known as 3D Gaussian Splatting (3D-GS) has attracted attention for its capability to achieve real-time rendering with high-quality results. By representing scenes with a collection of colored 3D Gaussians and employing a rasterization process, 3D-GS circumvents the need for computationally intensive volumetric rendering. This enables much faster image rendering rates, a crucial advancement for applications that require real-time performance.

Overcoming the Blurring Challenge

Despite its rendering speed, 3D-GS struggled to maintain image quality when the training images were blurred. Deblurring approaches designed for volumetric rendering-based methods weren't directly transferable to the rasterization-based 3D-GS. To address this, a novel deblurring framework adjusts the covariance matrices of 3D Gaussians through a small Multi-Layer Perceptron (MLP). By simulating the intermingling of neighboring pixels during training, the framework accurately models scene blurriness while retaining the ability to render images in real-time.

Enhancing 3D Scene Reconstruction

The paper introduces techniques to address sparse point clouds, another issue that arises when dealing with blurry images. By adding extra points with valid color features and pruning Gaussians based on their position, the process enhances the density of the point cloud representation, particularly in regions that traditional methods struggle with, like the far plane of a scene. As a result, the approach not only deblurs images but also reconstructs scenes with improved detail.

Achievements and Contributions

Empirically tested, the method demonstrates superior rendering quality or at least on par with other state-of-the-art models while boasting significantly greater rendering speeds (over 200 FPS). In doing so, it marks several contributions:

• The first real-time rendering-enabled defocus deblurring framework using 3D-GS
• A novel technique manipulating the covariance of each 3D Gaussian to model scene blurriness
• A training strategy that copes with sparse point clouds through calculated point addition and depth-based pruning
• State-of-the-art rendering quality achieved at unparalleled rendering speeds

Limitations and Future Prospects

Acknowledging limitations, the authors suggest that alternate deblurring methods based on volumetric rendering could potentially be adapted for rasterization-based 3D-GS. However, this might introduce additional computational costs. The paper concludes with the hope that future advancements will further refine deblurring techniques, potentially by developing grid blur kernels to address diverse types of real-world blurs while maintaining the rendering performance required for real-time applications.