MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervision (2410.19115v3)

Abstract: We present MoGe, a powerful model for recovering 3D geometry from monocular open-domain images. Given a single image, our model directly predicts a 3D point map of the captured scene with an affine-invariant representation, which is agnostic to true global scale and shift. This new representation precludes ambiguous supervision in training and facilitate effective geometry learning. Furthermore, we propose a set of novel global and local geometry supervisions that empower the model to learn high-quality geometry. These include a robust, optimal, and efficient point cloud alignment solver for accurate global shape learning, and a multi-scale local geometry loss promoting precise local geometry supervision. We train our model on a large, mixed dataset and demonstrate its strong generalizability and high accuracy. In our comprehensive evaluation on diverse unseen datasets, our model significantly outperforms state-of-the-art methods across all tasks, including monocular estimation of 3D point map, depth map, and camera field of view. Code and models can be found on our project page.

• The paper presents a novel direct geometry estimation method using affine-invariant point maps that eliminate focal-distance ambiguities in single-image depth recovery.
• It employs a robust training strategy with global ROE alignment and multi-scale local supervision, achieving a 35% reduction in estimation errors across open-domain images.
• The approach enhances practical applications by paving the way for advancements in 3D image editing, depth-to-image synthesis, and comprehensive scene understanding.

Monocular Geometry Estimation with MoGe

The paper "MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervision" presents a novel approach to 3D geometry recovery from single images, addressing a critical area in computer vision. The method, MoGe, distinguishes itself by predicting an affine-invariant 3D point map, a representation that is key to overcoming the inherent ambiguities of monocular estimation tasks.

Core Contribution

MoGe introduces a direct geometry estimation method utilizing affine-invariant point maps. Unlike previous models such as DUSt3R, which focus on scale-invariant representations for multi-view scenarios, MoGe effectively removes the focal-distance ambiguity. This is achieved through innovative training supervisions that enhance geometry learning.

Methodology

The model architecture is straightforward, directly mapping images to 3D point maps, from which depth maps and camera parameters can be derived. The use of affine-invariant point maps ensures the representation is free from global scale and shift uncertainties, facilitating robust training.

Key elements include:

1. Global and Local Supervision:
   • A robust, optimal, and efficient (ROE) alignment solver computes point cloud alignment, enhancing global shape learning.
   • A multi-scale local geometry loss addresses local geometric precision by employing independent affine alignments.
2. Training on Large-Scale Data: The model is trained on a diverse dataset corpus, demonstrating strong generalization abilities across open-domain images.

Numerical Results

The paper highlights MoGe's superior performance over existing methods in several benchmarks, achieving significant error reductions. For instance, MoGe achieved a 35% error reduction in monocular estimation tasks and over 20% in camera field of view predictions compared to the best previous approaches.

Implications

MoGe's contributions have far-reaching implications. By providing a reliable method for monocular geometry estimation, it paves the way for advancements in 3D-aware image editing, depth-to-image synthesis, and 3D scene understanding. Furthermore, it serves as a potent foundation model for further research in both video-based and multi-view 3D reconstruction.

Theoretical and Practical Impact

Theoretically, the affine-invariant representation and optimal supervision strategies introduced redefine existing approaches to address ambiguities in monocular tasks. Practically, the model's strong zero-shot performance across diverse datasets suggests it can be deployed across a variety of applications, enhancing the utility of monocular vision systems without requiring specialized training data or calibration procedures.

Future Directions

Looking ahead, integrating MoGe with other modalities, such as semantic segmentation and object recognition, could offer more comprehensive scene understanding capabilities. Moreover, expanding the model’s application to real-time systems might revolutionize fields requiring instantaneous 3D interpretation from single-view inputs.

In conclusion, MoGe represents a significant step forward in 3D geometry estimation, balancing innovation in training supervision with robust performance metrics. As the code and models are made available to the research community, they will likely spur further investigation and development in this vital area of computer vision.