Deblur-NeRF: Neural Radiance Fields from Blurry Images (2111.14292v2)

Abstract: Neural Radiance Field (NeRF) has gained considerable attention recently for 3D scene reconstruction and novel view synthesis due to its remarkable synthesis quality. However, image blurriness caused by defocus or motion, which often occurs when capturing scenes in the wild, significantly degrades its reconstruction quality. To address this problem, We propose Deblur-NeRF, the first method that can recover a sharp NeRF from blurry input. We adopt an analysis-by-synthesis approach that reconstructs blurry views by simulating the blurring process, thus making NeRF robust to blurry inputs. The core of this simulation is a novel Deformable Sparse Kernel (DSK) module that models spatially-varying blur kernels by deforming a canonical sparse kernel at each spatial location. The ray origin of each kernel point is jointly optimized, inspired by the physical blurring process. This module is parameterized as an MLP that has the ability to be generalized to various blur types. Jointly optimizing the NeRF and the DSK module allows us to restore a sharp NeRF. We demonstrate that our method can be used on both camera motion blur and defocus blur: the two most common types of blur in real scenes. Evaluation results on both synthetic and real-world data show that our method outperforms several baselines. The synthetic and real datasets along with the source code is publicly available at https://limacv.github.io/deblurnerf/

• The paper introduces the Deformable Sparse Kernel (DSK) module to jointly optimize blur kernels and radiance fields for enhanced image clarity.
• The method effectively mitigates both defocus and motion blur, outperforming traditional baselines in PSNR, SSIM, and LPIPS evaluations.
• This innovation broadens NeRF's real-world applicability and sets a new standard for handling non-ideal imaging conditions in neural rendering.

Deblur-NeRF: Enhancing NeRFs with Robustness to Image Blur

The paper "Deblur-NeRF: Neural Radiance Fields from Blurry Images" addresses a notable limitation in the application of Neural Radiance Fields (NeRF) for 3D scene reconstruction and novel view synthesis—specifically, the degradation caused by blur from defocus or motion. Through the introduction of Deblur-NeRF, the research pioneers a systematic approach to mitigate the effects of blur, demonstrating an enhanced capability to render sharp scenes from inherently blurry multi-view images.

NeRF has established itself as a powerful tool for scene reconstruction, utilizing a volumetric function parameterized by a multilayer perceptron (MLP) to map 3D locations and 2D directions to color and density outputs. However, the performance of NeRF deteriorates when the input images suffer from blurriness, resulting in artifacts and misaligned scene reconstructions. This paper is significant in its aim to enhance NeRF's robustness by developing a method that incorporates the simulation and modeling of the blur phenomenon directly into the rendering process.

Core Contributions

1. Deformable Sparse Kernel (DSK) Module: The novel component proposed by the authors is the DSK module, which dynamically models spatially-varying blur kernels through deformation of a canonical sparse kernel at each spatial location. Parameterized using an MLP, the DSK allows the joint optimization of the blur kernels along with the radiance fields.
2. Robustness to Blur: By integrating the blurred inputs in an analysis-by-synthesis framework, this method demonstrates significant improvements in dealing with both camera motion blur and defocus blur, outperforming several baselines in qualitative and quantitative evaluations.
3. Practical and Theoretical Implications: This advancement exhibits multiple implications for both practical applications and theoretical developments. Practically, it enhances the accuracy and aesthetic quality of NeRF-generated visualizations in real-world scenarios where capturing sharply focused images might be challenging. Theoretically, it pushes the boundaries of NeRF application under non-ideal conditions, setting a precedent for incorporating more sophisticated pre-processing and analysis mechanisms into neural scene representations.

Experimental Validation

The authors detail rigorous experimental evaluations, both on synthetic datasets and real-world scenarios, showcasing the effective handling of blur. The Deblur-NeRF demonstrates superior performance metrics such as PSNR, SSIM, and LPIPS compared to the naive approach of using blurry inputs directly or pre-deblurring using image-space techniques. This indicates its proficiency in maintaining multi-view consistency and achieving higher fidelity in the reconstructed views.

Future Directions

While the proposed method significantly enhances image quality, it may encounter challenges when faced with consistently applied blur patterns across scenes. Future advancements could explore incorporating learned image priors or adaptive perception training to further improve performance under uniform blur scenarios. Additionally, expanding the methodology to handle larger degrees of blur and other forms of visual artifacts could broaden its applicability.

In conclusion, this paper presents a substantial contribution to the field of neural rendering by skillfully integrating consideration for image blur into the NeRF framework, thus widening its applicability and setting the stage for future research in handling more complex visual variability.