GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image (2403.12013v1)

Abstract: We introduce GeoWizard, a new generative foundation model designed for estimating geometric attributes, e.g., depth and normals, from single images. While significant research has already been conducted in this area, the progress has been substantially limited by the low diversity and poor quality of publicly available datasets. As a result, the prior works either are constrained to limited scenarios or suffer from the inability to capture geometric details. In this paper, we demonstrate that generative models, as opposed to traditional discriminative models (e.g., CNNs and Transformers), can effectively address the inherently ill-posed problem. We further show that leveraging diffusion priors can markedly improve generalization, detail preservation, and efficiency in resource usage. Specifically, we extend the original stable diffusion model to jointly predict depth and normal, allowing mutual information exchange and high consistency between the two representations. More importantly, we propose a simple yet effective strategy to segregate the complex data distribution of various scenes into distinct sub-distributions. This strategy enables our model to recognize different scene layouts, capturing 3D geometry with remarkable fidelity. GeoWizard sets new benchmarks for zero-shot depth and normal prediction, significantly enhancing many downstream applications such as 3D reconstruction, 2D content creation, and novel viewpoint synthesis.

• The paper introduces GeoWizard, a generative model that jointly estimates depth and normals from a single image using diffusion priors.
• It employs a geometry switcher and scene distribution decoupler to ensure high fidelity and robust generalization across diverse scenes.
• Quantitative evaluations show that GeoWizard outperforms existing methods in zero-shot depth and normal estimation, setting new benchmarks.

Unveiling GeoWizard: A Generative Foundation Model for 3D Geometry Estimation from Single Images

Introduction

3D geometry estimation from single images is a pivotal challenge in computer vision, critical for numerous applications ranging from autonomous driving to content creation and beyond. The task, inherently ill-posed due to the loss of depth information in the projection process, has traditionally relied on discriminative models trained on specific datasets. These approaches, however, suffer from limitations in generalization and detail capture, primarily due to the diversity and quality constraints of available training data.

GeoWizard emerges as a novel paradigm in this landscape, proposing a generative foundation model that leverages the rich priors encapsulated within pre-trained diffusion models. By extending the stable diffusion model to jointly predict depth and surface normals, GeoWizard not only demonstrates superior generalization across diverse scenes but also excels in capturing intricate geometric details.

Key Contributions

• GeoWizard introduces a generative approach to the estimation of depth and normals from monocular images, showcasing remarkable generalization abilities and detail preservation.
• The model employs a geometry switcher within a unified framework for joint estimation, facilitating mutual information exchange between depth and normal predictions, thus ensuring high consistency between these geometric attributes.
• A novel strategy, termed the scene distribution decoupler, is proposed to handle the complex data distributions characteristic of varied scene layouts. This method significantly aids the model in distinguishing between different scene types, thereby improving the fidelity of 3D geometry estimation.

Methodology

GeoWizard's core relies on a modified diffusion model that encodes the diverse knowledge inherent in billions of images, thus benefiting depth and normal estimation tasks. A geometry switcher is employed to direct the model's focus either on depth or normals, underpinned by a shared generative process. This not only economizes on model parameters but also enhances geometric consistency through cross-domain self-attention mechanisms.

The scene distribution decoupler addresses the challenge of ambiguous geometric configurations arising from mixed scene layouts. By dissecting the overarching data distribution into sub-distributions representative of indoor, outdoor, and object-centric scenes, the model attains remarkable levels of fidelity and correctness in predicted depth and normals.

GeoWizard sets new benchmarks in zero-shot generalization for depth and normal estimation. Its capabilities extend to a variety of applications, including but not limited to, 3D reconstruction, content creation, and novel viewpoint synthesis, underscoring its potential as a foundational tool in computer vision and beyond.

Performance and Evaluation

Quantitative assessments underscore GeoWizard's superior performance across several benchmarks. In zero-shot evaluations involving depth estimation, the model consistently outperforms existing methods, reflecting its robustness and precision. Similarly, for surface normal estimation, GeoWizard demonstrates a keen ability to discern fine-grained details, outstripping current state-of-the-art solutions.

Future Work and Applications

GeoWizard illuminates the path forward for leveraging generative models in geometric estimation tasks. Future iterations could focus on enhancing efficiency, particularly in reducing the inference time through optimized diffusion steps. The fidelity and accuracy provided by GeoWizard open new avenues in 3D modeling, virtual reality, and augmented reality, offering tools of unprecedented power for creators and researchers alike.

Conclusion

GeoWizard represents a significant stride in the domain of 3D geometry estimation from single images. By harvesting the potential of generative models, specifically diffusion-based techniques, it introduces a novel, highly effective approach to understanding and reconstructing the three-dimensional world from two-dimensional inputs. Its inception marks a pivotal moment, promising to catalyze further innovations and applications in the fields of computer vision and digital content creation.