SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3D (2310.02596v2)

Abstract: It is inherently ambiguous to lift 2D results from pre-trained diffusion models to a 3D world for text-to-3D generation. 2D diffusion models solely learn view-agnostic priors and thus lack 3D knowledge during the lifting, leading to the multi-view inconsistency problem. We find that this problem primarily stems from geometric inconsistency, and avoiding misplaced geometric structures substantially mitigates the problem in the final outputs. Therefore, we improve the consistency by aligning the 2D geometric priors in diffusion models with well-defined 3D shapes during the lifting, addressing the vast majority of the problem. This is achieved by fine-tuning the 2D diffusion model to be viewpoint-aware and to produce view-specific coordinate maps of canonically oriented 3D objects. In our process, only coarse 3D information is used for aligning. This "coarse" alignment not only resolves the multi-view inconsistency in geometries but also retains the ability in 2D diffusion models to generate detailed and diversified high-quality objects unseen in the 3D datasets. Furthermore, our aligned geometric priors (AGP) are generic and can be seamlessly integrated into various state-of-the-art pipelines, obtaining high generalizability in terms of unseen shapes and visual appearance while greatly alleviating the multi-view inconsistency problem. Our method represents a new state-of-the-art performance with an 85+% consistency rate by human evaluation, while many previous methods are around 30%. Our project page is https://sweetdreamer3d.github.io/

Insightful Overview of "SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3D"

The paper "SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3D" explores the challenge of lifting 2D visuals derived from diffusion models into coherent 3D representations. Traditional 2D diffusion models are limited by their inherent lack of 3D awareness, often leading to inconsistencies across multiple views. This research identifies the primary root of this issue as geometric inconsistency and proposes a novel solution by aligning 2D geometric priors with defined 3D geometry during the transformation process.

The authors propose a method to fine-tune 2D diffusion models, making them viewpoint-aware to create view-specific coordinate maps. This process incorporates only coarse 3D information to resolve geometric inconsistencies while preserving the rich, detailed generation capabilities of 2D models. The key innovation is the creation of Aligned Geometric Priors (AGP), which provide a robust framework to mitigate multi-view inconsistencies and ensure high-quality, diversified output. This method has demonstrated a notable increase in consistency rate, achieving over 85% consistency according to human evaluation, which positions it well above the 30% benchmarks achieved by preceding methods.

Technical Approach and Methodology

1. Identifying Inconsistencies: The paper delineates two main types of inconsistencies in text-to-3D synthesis: geometric and appearance inconsistencies. Geometric inconsistencies are the major focus, as they more frequently result in perceptual errors when transitioning from 2D to 3D.
2. Geometric Priors in 2D Diffusion: The authors leverage the inherent geometric priors within 2D diffusion models, aligning these with 3D structures. By fine-tuning the 2D models to produce canonical coordinate maps that translate into 3D viewpoints, the method circumvents the traditional data-hungry requirements of 3D model training.
3. Utilizing Canonical Coordinates and Camera Conditioning: The process involves rendering depth maps from canonical 3D models, producing coordinate maps that serve as inputs during model fine-tuning. This integration of coarse yet consistent geometric information is computationally efficient and enhances the viewpoint-awareness needed for accurate 3D modeling.
4. Fine-tuning Procedures: Implementing model fine-tuning without compromising the generative capabilities of the original 2D model is crucial. This method allows the sweet spot of integrating coarse geometric alignment while maintaining the original model's capacity for high-fidelity and high-diversity output.

Integration and Results

The paper demonstrates the integration of AGP into existing state-of-the-art text-to-3D pipelines, such as DMTet-based and NeRF-based representations. The results highlight the versatility and compatibility of AGP, seamlessly enhancing geometric model accuracy without intruding upon appearance rendering. Quantitatively, this yielded a substantial improvement in the consistency rate, showcasing its efficacy above other contemporary approaches.

Implications and Future Directions

The implications of integrating AGP into text-to-3D systems extend to both theoretical and practical domains. Theoretically, the method capitalizes on the latent geometric knowledge embedded within 2D diffusion models, unlocking a more coherent path for translating 2D generative capabilities into the 3D field. Practically, this approach negates the pressing need for vast 3D datasets, cutting down on resource expenditure while maintaining output quality.

Looking forward, research could explore extending this alignment strategy towards addressing the rarer appearance inconsistencies, potentially through selective utilization of complementary appearance priors. Additionally, further refinement in handling complex geometric structures could open new avenues for more intricate and realistic 3D synthesis in real-time applications.

In conclusion, "SweetDreamer" delivers a critical advancement in overcoming the multi-view consistency challenges inherent in text-to-3D synthesis. By effectively realigning geometric priors within pre-trained 2D diffusion models, the paper elevates the potential and practical applicability of generative models across dimensional boundaries.