Dust to Tower: Coarse-to-Fine Photo-Realistic Scene Reconstruction from Sparse Uncalibrated Images (2412.19518v1)

Abstract: Photo-realistic scene reconstruction from sparse-view, uncalibrated images is highly required in practice. Although some successes have been made, existing methods are either Sparse-View but require accurate camera parameters (i.e., intrinsic and extrinsic), or SfM-free but need densely captured images. To combine the advantages of both methods while addressing their respective weaknesses, we propose Dust to Tower (D2T), an accurate and efficient coarse-to-fine framework to optimize 3DGS and image poses simultaneously from sparse and uncalibrated images. Our key idea is to first construct a coarse model efficiently and subsequently refine it using warped and inpainted images at novel viewpoints. To do this, we first introduce a Coarse Construction Module (CCM) which exploits a fast Multi-View Stereo model to initialize a 3D Gaussian Splatting (3DGS) and recover initial camera poses. To refine the 3D model at novel viewpoints, we propose a Confidence Aware Depth Alignment (CADA) module to refine the coarse depth maps by aligning their confident parts with estimated depths by a Mono-depth model. Then, a Warped Image-Guided Inpainting (WIGI) module is proposed to warp the training images to novel viewpoints by the refined depth maps, and inpainting is applied to fulfill the ``holes" in the warped images caused by view-direction changes, providing high-quality supervision to further optimize the 3D model and the camera poses. Extensive experiments and ablation studies demonstrate the validity of D2T and its design choices, achieving state-of-the-art performance in both tasks of novel view synthesis and pose estimation while keeping high efficiency. Codes will be publicly available.

• The paper presents a novel two-stage method (D2T) that first generates a coarse 3D model and then refines it with depth alignment and image-guided inpainting.
• It bypasses the need for pre-computed camera parameters by leveraging sparse-view techniques and SfM-free strategies.
• D2T achieves state-of-the-art rendering quality and pose estimation rapidly, demonstrating strong potential for real-time 3D reconstruction applications.

Coarse-to-Fine Photo-Realistic Scene Reconstruction from Sparse Uncalibrated Images

The paper "Dust to Tower: Coarse-to-Fine Photo-Realistic Scene Reconstruction from Sparse Uncalibrated Images" addresses the challenge of generating accurate 3D scene reconstructions from a minimal number of images without requiring pre-computed camera parameters. This is particularly pertinent in computer vision tasks such as augmented reality and autonomous navigation, where accurate scene understanding is vital. The proposed method, Dust to Tower (D2T), leverages both sparse-view and SfM-free methodologies to optimize 3D Gaussian Splatting (3DGS) and camera poses from sparse uncalibrated images, presenting a significant advancement in efficient scene reconstruction.

Methodology Overview

D2T employs a two-stage strategy for scene reconstruction: a coarse construction stage followed by a refinement stage.

1. Coarse Construction: The method initiates by constructing a coarse 3D model from sparse images using a Multi-View Stereo model, DUSt3R, which efficiently computes both an initial 3D point cloud and camera poses. This eliminates the need for the extensive calculation typically required by Structure-from-Motion (SfM) methods and provides a fast, albeit rough, baseline for further refinement.
2. Refinement via Warping and Inpainting: The refinement stage introduces two novel modules:
   • Confidence Aware Depth Alignment (CADA): This module enhances the quality of depth maps by aligning a high-precision monocular depth estimation with coarse depth data, refining the depth maps that are crucial for accurate image warping.
   • Warped Image-Guided Inpainting (WIGI): Using the refined depth maps, this module warps images to unobserved viewpoints and fills in the missing data (due to changes in viewpoint) through efficient inpainting. This enriched data provides additional constraints that help refine the 3D model and pose estimations, mitigating overfitting to the sparse initial dataset.

Numerical Results and Efficiency

Experimentation across various datasets, including Tanks and Temples, MipNeRF360, and CO3D V2, demonstrates that D2T achieves state-of-the-art performance in terms of rendering quality and pose estimation accuracy, outperforming existing methods significantly. The efficiency of the proposed methodology is notable, with the reconstruction process from sparse views completed in seconds, as opposed to several hours required by comparable methodologies. This is a substantial improvement, indicating the method’s viability for real-time applications.

Implications and Future Scope

The implications of this research are significant, especially in fields requiring rapid, scalable 3D reconstruction from limited input data, such as robotics and interactive media. The paper suggests that the combination of efficient depth refinement and high-fidelity inpainting can bridge the gap between sparse data availability and the need for high-quality scene rendering.

Future work could explore the extension of D2T to even larger-scale scenes or objects, particularly in cases involving dynamic elements or complex lighting conditions. Moreover, integrating learning-based methods for further optimizing the depth alignment and inpainting processes holds potential for enhancing both speed and accuracy. As AI-driven imaging continues to evolve, the methodologies developed in this paper provide a robust framework for future advances in 3D scene reconstruction technology.