Unifying Correspondence, Pose and NeRF for Pose-Free Novel View Synthesis from Stereo Pairs (2312.07246v2)

Abstract: This work delves into the task of pose-free novel view synthesis from stereo pairs, a challenging and pioneering task in 3D vision. Our innovative framework, unlike any before, seamlessly integrates 2D correspondence matching, camera pose estimation, and NeRF rendering, fostering a synergistic enhancement of these tasks. We achieve this through designing an architecture that utilizes a shared representation, which serves as a foundation for enhanced 3D geometry understanding. Capitalizing on the inherent interplay between the tasks, our unified framework is trained end-to-end with the proposed training strategy to improve overall model accuracy. Through extensive evaluations across diverse indoor and outdoor scenes from two real-world datasets, we demonstrate that our approach achieves substantial improvement over previous methodologies, especially in scenarios characterized by extreme viewpoint changes and the absence of accurate camera poses.

• The paper introduces CoPoNeRF, a unified framework that integrates correspondence estimation, pose prediction, and neural rendering from stereo pairs.
• It employs multi-level feature maps and 4D cost volumes to align features efficiently and boost pose accuracy with attention-based rendering.
• Evaluation on diverse datasets demonstrates significant improvements in novel view synthesis, especially under extreme viewpoint variations and limited overlap.

Unifying Pose Estimation and Neural Rendering from Stereo Images

Introduction

Traditionally, generating new views from stereo images involves estimating camera poses with pre-existing tools and fusing those with neural radiance fields (NeRF) models to synthesize the view. This separation of tasks can lead to inaccuracies due to misalignments and disparities. Recognizing the mutual dependencies within these tasks of 2D correspondence, camera pose estimation, and NeRF rendering, a new framework named CoPoNeRF is introduced, which integrates these functionalities to create enhance 3D geometric understandings from stereo pairs, even without known camera poses.

Approach and Framework

CoPoNeRF stands out by employing a shared network representation that serves multiple components, each responsible for a different part of the view synthesis procedure. This shared approach where correspondence estimation, pose, and rendering inform one another allows for collective enhancement.

The method begins by extracting multi-level feature maps, which are then utilized to build comprehensive 4D cost volumes for correspondence estimation. These volumes aid in the extraction of flow and relative camera pose between two views. Importantly, the cost volumes double as matching distributions to align features efficiently, improving pose prediction. The renderer then uses these estimations to synthesize the novel view by leveraging an attention-based rendering procedure.

The framework's abilities are cemented by a training strategy that includes image reconstruction, matching, pose, and triplet consistency losses. The latter assesses the consistency across depth and optical flow estimations, reinforcing the interrelated accuracy of separate outputs.

Evaluation and Results

CoPoNeRF's efficacy is benchmarked on large-scale indoor and outdoor datasets, where the method is shown to excel in rendering quality and pose estimation, particularly in scenarios with extreme viewpoint changes and limited overlap. It outperforms existing methods that treat pose estimation and NeRF rendering as separate stages or utilize staged training. Additionally, ablation studies confirm that each component of the CoPoNeRF pipeline contributes meaningfully to the overall performance.

Impact and Future Work

The unification of correspondence, pose, and NeRF within CoPoNeRF signifies a significant stride toward practical and accurate novel view synthesis from stereo pairs. By jointly optimizing these interdependent tasks, the framework achieves an enhanced understanding of 3D geometry and robustness against variable conditions. Future work may involve extending the CoPoNeRF principles to even more challenging data scenarios, continuous refinement of the shared representation, and further exploration of how intricate network components together improve the outcome of joint estimations.