IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation (2402.08682v1)

Abstract: Most text-to-3D generators build upon off-the-shelf text-to-image models trained on billions of images. They use variants of Score Distillation Sampling (SDS), which is slow, somewhat unstable, and prone to artifacts. A mitigation is to fine-tune the 2D generator to be multi-view aware, which can help distillation or can be combined with reconstruction networks to output 3D objects directly. In this paper, we further explore the design space of text-to-3D models. We significantly improve multi-view generation by considering video instead of image generators. Combined with a 3D reconstruction algorithm which, by using Gaussian splatting, can optimize a robust image-based loss, we directly produce high-quality 3D outputs from the generated views. Our new method, IM-3D, reduces the number of evaluations of the 2D generator network 10-100x, resulting in a much more efficient pipeline, better quality, fewer geometric inconsistencies, and higher yield of usable 3D assets.

• The paper introduces IM-3D, which leverages text-conditioned video diffusion and Gaussian splatting to achieve rapid, high-quality 3D asset generation.
• It employs an iterative refinement process that minimizes geometric inconsistencies and reduces processing time from hours to approximately 3 minutes.
• The approach circumvents limitations of traditional SDS-based methods by generating consistent multi-view images for robust 3D reconstruction.

Enhancing 3D Content Generation through IM-3D: Iterative Multiview Diffusion and Reconstruction

Introduction to IM-3D

Current approaches in text-to-3D generation heavily rely on 2D generators trained on large-scale image data, due to the scarcity of high-quality 3D data. However, these methods, including those based on Score Distillation Sampling (SDS) and its variants, are characterized by limitations such as slowness, instability, and the propensity for artifacts. The IM-3D (Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation) technique introduces a novel methodology that significantly improves upon these issues by adopting a faster, more efficient pipeline capable of producing high-yield, high-quality 3D assets with fewer geometric inconsistencies.

Methodology Overview

IM-3D constitutes a shift from conventional text-to-image models towards utilizing video diffusion models, specifically the Emu Video model, which is conditioned both on text and reference images. This method enables the generation of consistent views of an object, effectively forming a 360° video around it, thereby facilitating robust 3D reconstruction without necessitating extensive SDS or sophisticated reconstruction networks. The core contributions of this approach include:

• Generating high-resolution, consistent multi-view images using a text-to-video model, thereby reducing the requirement for substantial SDS evaluations.
• A Gaussian splatting-based 3D reconstruction algorithm that leverages robust image-based losses for enhanced quality and speed.
• An iterative refinement process that further enhances model consistency and detail by integrating feedback loops between the 2D generator and 3D reconstruction.

Technical Insights and Innovations

The implementation of IM-3D showcases several technical advancements, including the fine-tuning of the Emu Video model for generating high-quality, consistent multi-view videos that serve as a basis for 3D object reconstruction. The reconstruction process employs Gaussian splatting, which offers a blend of efficiency and fidelity, supported by image-level losses like LPIPS for addressing small inconsistencies. This approach circumvents common pitfalls associated with SDS, such as artifacts and low diversity, by reducing the requisite number of evaluations from tens of thousands to merely around 40, with further reductions in iterated generations.

Comparative Analysis and Results

IM-3D demonstrates superior performance in terms of both efficiency and output quality when compared to existing methods. Particularly notable is its ability to maintain high visual faithfulness with significantly reduced processing times (approximately 3 minutes for a complete asset generation cycle), representing a substantial improvement over the hour-long durations typical of other models. Comparative evaluations underscored IM-3D’s advancements in generating 3D models that exhibit remarkable fidelity to both textual and visual prompts.

Practical Implications and Future Directions

The introduction of IM-3D marks a significant advancement in the field of 3D content generation, offering a more scalable, efficient, and high-quality alternative to existing methodologies. Its ability to directly leverage video diffusion models for creating consistent multi-view images revolutionizes the text-to-3D conversion process, opening new avenues for research and application. Future explorations could focus on further refining the model’s efficiency and expanding its applicability across a broader spectrum of 3D content creation tasks, potentially setting new benchmarks in the domain.

Conclusion

IM-3D represents a pivotal shift towards more effective and higher-quality 3D content generation from textual and image inputs. By addressing and mitigating the limitations inherent to previous methods, it not only enhances the efficiency and quality of generated 3D models but also paves the way for future innovations in the field. The methodology’s emphasis on video-based multi-view generation and robust 3D reconstruction algorithms sets a new standard for what is achievable in text-to-3D conversions, holding promise for a wide array of applications in digital content creation and beyond.