Dense Depth Priors for Neural Radiance Fields from Sparse Input Views (2112.03288v2)

Abstract: Neural radiance fields (NeRF) encode a scene into a neural representation that enables photo-realistic rendering of novel views. However, a successful reconstruction from RGB images requires a large number of input views taken under static conditions - typically up to a few hundred images for room-size scenes. Our method aims to synthesize novel views of whole rooms from an order of magnitude fewer images. To this end, we leverage dense depth priors in order to constrain the NeRF optimization. First, we take advantage of the sparse depth data that is freely available from the structure from motion (SfM) preprocessing step used to estimate camera poses. Second, we use depth completion to convert these sparse points into dense depth maps and uncertainty estimates, which are used to guide NeRF optimization. Our method enables data-efficient novel view synthesis on challenging indoor scenes, using as few as 18 images for an entire scene.

• The paper introduces a method that leverages sparse-to-dense depth conversion and uncertainty estimates to constrain NeRF optimization.
• The approach integrates depth-guided sampling focused on predicted scene surfaces, improving convergence and rendering quality.
• Empirical results demonstrate high-quality view synthesis with enhanced geometric fidelity using merely 18–36 images for room scenes.

Dense Depth Priors for Neural Radiance Fields from Sparse Input Views

The paper by Roessle et al. presents a method to enhance the data efficiency of novel view synthesis using Neural Radiance Fields (NeRF) by leveraging dense depth priors. This work addresses the challenge of synthesizing realistic views for room-sized scenes from significantly fewer images than traditionally required by NeRF, which typically depends on hundreds of static input views.

Problem and Motivation

NeRF has demonstrated significant success in creating photorealistic novel views by representing scenes using a neural network, but its dependency on extensive input data poses limitations in real-world applications, particularly for room-scale scenes with inherent viewing challenges like minimal texture, color inconsistencies, and sparse image overlaps. The paper tackles this limitation by integrating depth priors to constrain the optimization of NeRF, enabling high-quality view synthesis from a minimal set of images.

Proposed Method

The proposed solution involves three key components:

1. Sparse-to-Dense Depth Conversion: The authors utilize available sparse depth data from the structure from motion (SfM) process to inform their approach. They employ a depth completion network to convert sparse depth points into dense depth maps with associated uncertainty estimates. This step is crucial for providing detailed geometric information where direct image correspondences may be weak or ambiguous due to sparse overlaps or textureless surfaces.
2. Depth-Guided NeRF Optimization: The dense depth maps and their uncertainty estimates guide the NeRF optimization process. By imposing these as constraints, the optimization incorporates additional geometric fidelity into the learning process. This is achieved by supervising the NeRF with a loss function sensitive to the depth prior, balancing between fitting the observed images and respecting the provided geometric guidance.
3. Depth-Guided Sampling: The sampling strategy during both training and inference stages is modified to exploit the dense depth information, focusing sample points around predicted scene surfaces, thereby improving convergence and rendering quality without the computational burden of coarse sampling grids.

Results and Evaluation

Empirically, the authors demonstrate that their approach enables NeRF to effectively synthesize high-quality novel views using as few as 18--36 images for entire room scenes. Quantitative evaluations on datasets such as Matterport3D and ScanNet indicate improvements over traditional NeRF and other baselines, particularly in view synthesis quality and geometric consistency as measured by root mean square error (RMSE) in depth estimation.

Implications and Future Directions

The integration of dense depth priors effectively reduces the data requirements for NeRF, broadening its applicability in practical scenarios such as virtual reality where quick and inexpensive data capture is crucial. The method demonstrates resilience to noisy and incomplete depth data, offering robust and detailed scene reconstructions. Future research may explore the development of more generic depth completion methods that do not rely on extensive pre-training on large datasets or investigate the application of similar techniques to dynamic scenes and outdoor environments.

In conclusion, this research marks a significant advance in the practical application of NeRF for real-world scenarios, showcasing the potential of combining photometric data with geometric priors to overcome the longstanding challenges of data efficiency in scene synthesis.