UVRM: A Scalable 3D Reconstruction Model from Unposed Videos (2501.09347v2)

Abstract: Large Reconstruction Models (LRMs) have recently become a popular method for creating 3D foundational models. Training 3D reconstruction models with 2D visual data traditionally requires prior knowledge of camera poses for the training samples, a process that is both time-consuming and prone to errors. Consequently, 3D reconstruction training has been confined to either synthetic 3D datasets or small-scale datasets with annotated poses. In this study, we investigate the feasibility of 3D reconstruction using unposed video data of various objects. We introduce UVRM, a novel 3D reconstruction model capable of being trained and evaluated on monocular videos without requiring any information about the pose. UVRM uses a transformer network to implicitly aggregate video frames into a pose-invariant latent feature space, which is then decoded into a tri-plane 3D representation. To obviate the need for ground-truth pose annotations during training, UVRM employs a combination of the score distillation sampling (SDS) method and an analysis-by-synthesis approach, progressively synthesizing pseudo novel-views using a pre-trained diffusion model. We qualitatively and quantitatively evaluate UVRM's performance on the G-Objaverse and CO3D datasets without relying on pose information. Extensive experiments show that UVRM is capable of effectively and efficiently reconstructing a wide range of 3D objects from unposed videos.

• The paper introduces UVRM, a novel 3D reconstruction method that eliminates the need for camera pose annotations.
• It employs a transform-based network with score distillation sampling and analysis-by-synthesis to generate pose-invariant latent features.
• UVRM demonstrates superior speed and accuracy over traditional methods on both synthetic and real-world datasets.

UVRM: A Scalable 3D Reconstruction Model from Unposed Videos

This paper presents a novel approach to 3D reconstruction referred to as the Unposed Video Reconstruction Model (UVRM). This approach represents a notable advancement in the field of 3D computer vision, primarily addressing the constraint of requiring prior camera pose knowledge during training. By rethinking the conventional 3D reconstruction pipeline, UVRM allows for the training of 3D models without relying on camera pose information from monocular video data.

Methodological Contributions

The authors introduce UVRM as a significant stride towards the development of scalable 3D models trained directly from video data. Key to this model is a transform-based network architecture that aggregates video frames into a pose-invariant latent feature space. The UVRM pipeline subsequently decodes this latent space into a tri-plane 3D representation, facilitating the 3D reconstruction process.

To circumvent the requirement for pose annotations, UVRM utilizes two primary innovations: Score Distillation Sampling (SDS) and an analysis-by-synthesis framework. Through SDS, UVRM is empowered to synthesize pseudo novel views of objects via a pretrained diffusion model, leading to qualitative improvements in pose-agnostic scenarios. The analysis-by-synthesis approach further strengthens the training process by iteratively augmenting the inputs with view-consistent samples created from these generated novel views.

Empirical Results

UVRM has been comprehensively evaluated on datasets such as the synthetic G-Objaverse and the real-world CO3D, showing its applicability and robustness across diverse scenarios. Importantly, experiments demonstrated UVRM's capacity to produce high-quality 3D reconstructions from unposed videos, outperforming traditional methods that require pose information.

The results from these experiments highlight UVRM's capability of performing accurate 3D reconstructions with improved speed when generalized across multiple objects simultaneously. This is in stark contrast to standard optimization-based methods that typically demand extensive computational resources.

Theoretical and Practical Implications

The paper's contributions have significant implications for 3D computer vision and related fields. Theoretically, by obviating the need for pose information, UVRM challenges the long-standing paradigm in 3D reconstruction that is reliant on accurately annotated pose data. This innovation could pave the way for more robust volumetric modeling systems that are less sensitive to errors in pose estimation.

From a practical perspective, UVRM sets a precedent for utilizing the abundance of video data available today, bypassing the traditional bottleneck of synthetic datasets. This offers substantial scalability for applications in diverse domains such as augmented reality, virtual simulations, and automated design systems.

Future Developments

The methodological insights from UVRM suggest several potential directions for future research in AI and computer vision:

1. Extending UVRM: Further refining the robustness of UVRM in diverse and challenging real-world scenarios, potentially integrating multi-modal data sources to enrich the reconstruction quality.
2. Large-Scale Deployment: Leveraging UVRM's pose-free framework to scale up and utilize vast video datasets for training even more generalizable large reconstruction models (LRMs).
3. Cross-Domain Applications: Exploring UVRM’s utility in broader contexts, such as its implementation in robotics for environment perception or in film and gaming industries for automatic 3D scene generation.

In summary, UVRM represents a significant technical contribution to the domain of 3D reconstruction, offering a scalable solution to 3D modeling challenges traditionally limited by the need for precise camera pose data. Its integration of innovative methods such as SDS and analysis-by-synthesis marks a step forward in the pursuit of creating expansive and versatile 3D foundation models.