BAD-Gaussians: Bundle Adjusted Deblur Gaussian Splatting (2403.11831v2)

Abstract: While neural rendering has demonstrated impressive capabilities in 3D scene reconstruction and novel view synthesis, it heavily relies on high-quality sharp images and accurate camera poses. Numerous approaches have been proposed to train Neural Radiance Fields (NeRF) with motion-blurred images, commonly encountered in real-world scenarios such as low-light or long-exposure conditions. However, the implicit representation of NeRF struggles to accurately recover intricate details from severely motion-blurred images and cannot achieve real-time rendering. In contrast, recent advancements in 3D Gaussian Splatting achieve high-quality 3D scene reconstruction and real-time rendering by explicitly optimizing point clouds as Gaussian spheres. In this paper, we introduce a novel approach, named BAD-Gaussians (Bundle Adjusted Deblur Gaussian Splatting), which leverages explicit Gaussian representation and handles severe motion-blurred images with inaccurate camera poses to achieve high-quality scene reconstruction. Our method models the physical image formation process of motion-blurred images and jointly learns the parameters of Gaussians while recovering camera motion trajectories during exposure time. In our experiments, we demonstrate that BAD-Gaussians not only achieves superior rendering quality compared to previous state-of-the-art deblur neural rendering methods on both synthetic and real datasets but also enables real-time rendering capabilities. Our project page and source code is available at https://lingzhezhao.github.io/BAD-Gaussians/

• The paper presents BAD-Gaussians, a method that jointly optimizes 3D Gaussian representations and camera trajectories to reconstruct deblurred scenes.
• It utilizes photometric bundle adjustment and spline-based interpolation to synthesize virtual sharp images that guide accurate optimization.
• Experimental results on synthetic and real datasets show superior deblurring, novel view synthesis, and real-time rendering over existing methods.

Exploring Bundle Adjusted Deblur Gaussian Splatting for Enhanced Scene Reconstruction from Motion-Blurred Images

Introduction to BAD-Gaussians

Recent developments have introduced a novel method named Bundle Adjusted Deblur Gaussian Splatting (BAD-Gaussians), addressing the challenge of reconstructing high-quality 3D scenes from motion-blurred images with inaccurate camera poses. The method is based on explicit 3D Gaussian Splatting, which outperforms traditional Neural Radiance Fields (NeRF) methods in handling severely blurred images. The approach models the physical image formation process of motion-blurred images and jointly learns the parameters of Gaussians while recovering camera motion trajectories during exposure time. This results in superior rendering quality and the ability to achieve real-time rendering capabilities.

Rationale and Methodology

Leveraging 3D Gaussian Splatting

BAD-Gaussians employ 3D Gaussian Splatting (3D-GS) to explicitly optimize point clouds as Gaussian spheres for high-quality 3D scene reconstruction and real-time rendering. The method overcomes the limitations of NeRF and 3D-GS, which struggle with motion-blurred images and inaccurate camera poses, by utilizing explicit Gaussian representation to address these issues. The approach presents a significant improvement in both efficiency and quality over implicit methods.

Handling Motion-Blurred Images

The method introduces a photometric bundle adjustment formulation tailored for motion-blurred images. It interpolates camera motion trajectories using a spline function, generating virtual sharp images by projecting scene Gaussians onto the image plane. By averaging these virtual images, the method synthesizes blurred images following the physical blur process. The optimization involves minimizing the photometric error between synthesized and input blurred images through differentiable Gaussian rasterization.

Experimental Validation

BAD-Gaussians was validated using synthetic and real datasets, including the challenging MBA-VO dataset characterized by severe motion blur and variable camera velocities. The method demonstrated remarkable performance improvements in deblurring and novel view synthesis tasks compared to state-of-the-art methods. Specifically, it surpassed the performance of previous implicit neural rendering methods by incorporating the image formation process of motion-blurred images, achieving both high-quality rendering and real-time rendering speed.

Pose Estimation Performance

The method also showed superior accuracy in recovering camera motion trajectories, outperforming classical structure-from-motion frameworks and previous deblurring rendering strategies. This highlights BAD-Gaussians' ability to accurately model the camera motion during image exposure, contributing to its enhanced scene reconstruction capabilities.

Implications and Future Directions

BAD-Gaussians' introduction marks a significant advancement in the field of 3D scene reconstruction from motion-blurred images. By efficiently addressing the challenges posed by motion blur and inaccurate camera poses, the method opens new possibilities for applications requiring high-fidelity 3D models from imperfect inputs, such as virtual and augmented reality, robotic navigation, and more. Future developments may explore further optimizations in trajectory modeling and the extension of this methodology to other types of image distortions, broadening the scope of practical applications.