BAD-NeRF: Bundle Adjusted Deblur Neural Radiance Fields (2211.12853v2)

Abstract: Neural Radiance Fields (NeRF) have received considerable attention recently, due to its impressive capability in photo-realistic 3D reconstruction and novel view synthesis, given a set of posed camera images. Earlier work usually assumes the input images are of good quality. However, image degradation (e.g. image motion blur in low-light conditions) can easily happen in real-world scenarios, which would further affect the rendering quality of NeRF. In this paper, we present a novel bundle adjusted deblur Neural Radiance Fields (BAD-NeRF), which can be robust to severe motion blurred images and inaccurate camera poses. Our approach models the physical image formation process of a motion blurred image, and jointly learns the parameters of NeRF and recovers the camera motion trajectories during exposure time. In experiments, we show that by directly modeling the real physical image formation process, BAD-NeRF achieves superior performance over prior works on both synthetic and real datasets. Code and data are available at https://github.com/WU-CVGL/BAD-NeRF.

• The paper introduces BAD-NeRF, which jointly optimizes NeRF parameters and camera motion trajectories to robustly handle motion blur in 3D scene reconstruction.
• It employs a linear motion model in SE(3) to accurately capture exposure-time camera movements, directly integrating blur into the NeRF training pipeline.
• Empirical results show improved PSNR, SSIM, and LPIPS metrics over existing methods, leading to better deblurring performance and novel view synthesis.

BAD-NeRF: Bundle Adjusted Deblur Neural Radiance Fields

The paper entitled "BAD-NeRF: Bundle Adjusted Deblur Neural Radiance Fields" presents an innovative solution to the problem of incorporating motion blurred images into Neural Radiance Fields (NeRF) frameworks, a topic of paramount interest in computer vision due to the growing application of NeRFs in 3D reconstruction and novel view synthesis. The researchers introduce the BAD-NeRF methodology, which jointly learns the parameters of NeRF and recovers camera motion trajectories during exposure time, offering a significant enhancement over previous methodologies that generally assume sharp input images.

Problem Context and Motivation

Traditional approaches in 3D scene reconstruction often rely on the assumption of high-quality, sharp images. The advent of NeRFs allowed for impressive 3D representations and view synthesis from multi-view posed images. However, scenarios with image degradations such as motion blur present a challenge for these approaches. Motion blur is especially prevalent in low-light conditions or high-speed camera motion, leading to degraded performance in NeRF training. This paper addresses the critical assumption violation within NeRF paradigms by proposing a framework that accounts for motion blur, enhancing training robustness and output quality.

Methodology

BAD-NeRF innovatively integrates the physical process of motion blur image formation into the NeRF training pipeline. The framework models the blur as a result of camera motion during exposure, and optimally estimates both camera motion trajectories and the NeRF parameters. Key to this approach is the linear motion model in SE(3) space for representing camera trajectory, which, within the typically short exposure times, assumes linear camera movement without compromising accuracy.

The methodological foundation of BAD-NeRF revolves around:

• Joint Optimization: Simultaneously optimizing the NeRF parameters and camera poses during the exposure period, overcoming limitations from inaccurate initial camera position estimates.
• Exposure-Time Motion Modeling: Utilizing a linear model to interpolate poses in SE(3) space, this formulation captures the trajectory of the camera during the motion blur, facilitating accurate rendering and synthesis.

The articulated photo-metric bundle adjustment formulation under the NeRF framework allows a direct simulation of the image formation process with motion blur, advancing state-of-the-art performance in both deblurring and synthesis tasks.

Empirical Evaluation and Results

The empirical assessment conducted on both synthetic datasets and real-world representations from prior research, including datasets from Deblur-NeRF and MBA-VO, underscores the efficacy of BAD-NeRF. Quantitative metrics such as PSNR, SSIM, and LPIPS demonstrate superior performance over several contemporary deblurring techniques, both single image and multi-view based.

• Deblurring Performance: BAD-NeRF significantly outperforms rivals like SRNDeblurNet and Deblur-NeRF, especially in scenarios with severe image motion blur. Notably, BAD-NeRF maintains robust performance regardless of the blur level, highlighting its adaptability.
• Novel View Synthesis: Achieving high-quality novel view renderings with better image metrics by addressing occlusions effectively.
• Pose Estimation: Improved accuracy in camera pose estimation compared to methods such as BARF, facilitated by explicit trajectory modeling.

Implications and Future Directions

The practical implications of BAD-NeRF are substantial, providing a reliable framework for applications requiring precise 3D rendering from degraded image sets, such as autonomous navigation and augmented reality. Theoretically, this work paves the way for further enhancements in NeRF robustness and versatility, including potential integrations with dynamic scene or non-rigid object reconstructions.

Future advancements could explore more complex camera motion models or adaptive time-dependent trajectories to capture even more intricate movements, possibly extending BAD-NeRF’s utility in more varied environments and broader deployment scenarios. Additionally, exploring integration with other visual SLAM and tracking systems could further enhance its practical application.

In conclusion, BAD-NeRF charts a forward path in the handling of motion blurred images within NeRF systems, effectively bridging the gap between theoretical robustness and practical applicability, and opening myriad possibilities for both academic inquiry and real-world deployment.