Local-to-Global Registration for Bundle-Adjusting Neural Radiance Fields (2211.11505v3)

Abstract: Neural Radiance Fields (NeRF) have achieved photorealistic novel views synthesis; however, the requirement of accurate camera poses limits its application. Despite analysis-by-synthesis extensions for jointly learning neural 3D representations and registering camera frames exist, they are susceptible to suboptimal solutions if poorly initialized. We propose L2G-NeRF, a Local-to-Global registration method for bundle-adjusting Neural Radiance Fields: first, a pixel-wise flexible alignment, followed by a frame-wise constrained parametric alignment. Pixel-wise local alignment is learned in an unsupervised way via a deep network which optimizes photometric reconstruction errors. Frame-wise global alignment is performed using differentiable parameter estimation solvers on the pixel-wise correspondences to find a global transformation. Experiments on synthetic and real-world data show that our method outperforms the current state-of-the-art in terms of high-fidelity reconstruction and resolving large camera pose misalignment. Our module is an easy-to-use plugin that can be applied to NeRF variants and other neural field applications. The Code and supplementary materials are available at https://rover-xingyu.github.io/L2G-NeRF/.

• The paper introduces L2G-NeRF, a novel two-step method that combines pixel-wise local alignment with frame-wise global registration to address camera pose inaccuracies.
• It leverages deep networks for flexible pixel alignment and differentiable solvers for global transformation, enhancing novel view synthesis quality.
• Experimental results on synthetic and real-world datasets demonstrate significant improvements over state-of-the-art methods in metrics like PSNR, SSIM, and LPIPS.

An Overview of Local-to-Global Registration for Bundle-Adjusting Neural Radiance Fields

The paper "Local-to-Global Registration for Bundle-Adjusting Neural Radiance Fields" addresses the challenge of registering camera frames and reconstructing neural fields simultaneously, a core problem within the field of photometric bundle adjustment (BA). This research introduces L2G-NeRF, a novel framework that enhances the registration and reconstruction capabilities of Neural Radiance Fields (NeRF) through a local-to-global registration strategy.

Key Contributions and Methodology

L2G-NeRF builds upon the limitations of existing NeRF methods, particularly their dependence on accurate camera poses for high-fidelity novel view synthesis. The paper tackles the instability in jointly optimizing neural 3D representations and camera frame registration, which tends to result in suboptimal solutions due to poor initialization.

The authors propose a two-step process combining pixel-wise local alignment and frame-wise global alignment:

1. Pixel-wise Local Alignment: This initial non-parametric stage involves learning pixel-wise flexible transformations using a deep network that minimizes photometric reconstruction errors. This allows the method to start with a flexible, unsupervised pixel-level correction.
2. Frame-wise Global Alignment: Following the local alignment, a parametric approach is employed by introducing differentiable parameter estimation solvers. These solvers leverage whole-frame correspondences to compute global transformations, acting as a geometric constraint for localized transformations.

Experiments conducted with both synthetic and real-world datasets demonstrate that L2G-NeRF surpasses the current state-of-the-art in terms of both camera pose registration and reproduction quality. Notably, the method shows robustness in environments exhibiting large camera pose misalignments and can be easily integrated as a plugin into existing NeRF systems and other neural field applications.

Numerical Results and Claims

The paper provides comprehensive numerical results showcasing L2G-NeRF's superior performance compared to existing methods like BARF. Quantitative evaluations based on metrics like rotation and translation errors, PSNR, SSIM, and LPIPS establish that L2G-NeRF achieves accurate camera alignment and superior image synthesis quality.

The authors also showcase compelling qualitative results in visualizing 3D scene reconstructions, highlighting the effectiveness of L2G-NeRF's novel local-to-global registration process. This facilitates applications where robust visual computing is critical.

Implications and Future Directions

The implications of this work are far-reaching for photorealistic rendering and computer vision applications requiring accurate 3D scene reconstruction from images with uncalibrated camera poses. The research advances the field by offering a robust alternative to traditional BA methods, facilitating their application in dynamic and real-world environments.

However, the paper acknowledges limitations such as the inability to recover camera poses from scratch under inward-facing 360-degree scenes. Future research directions could consider integrating additional methods like epipolar geometry and graph optimization to handle these challenges.

In conclusion, this paper represents a significant advancement in the field of neural field registration, contributing a pragmatic solution applicable across a spectrum of visual computing tasks. The availability of the authors' code and supplementary materials promises to aid further research and adoption within the community.