Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors (2306.17843v2)

Abstract: We present Magic123, a two-stage coarse-to-fine approach for high-quality, textured 3D meshes generation from a single unposed image in the wild using both2D and 3D priors. In the first stage, we optimize a neural radiance field to produce a coarse geometry. In the second stage, we adopt a memory-efficient differentiable mesh representation to yield a high-resolution mesh with a visually appealing texture. In both stages, the 3D content is learned through reference view supervision and novel views guided by a combination of 2D and 3D diffusion priors. We introduce a single trade-off parameter between the 2D and 3D priors to control exploration (more imaginative) and exploitation (more precise) of the generated geometry. Additionally, we employ textual inversion and monocular depth regularization to encourage consistent appearances across views and to prevent degenerate solutions, respectively. Magic123 demonstrates a significant improvement over previous image-to-3D techniques, as validated through extensive experiments on synthetic benchmarks and diverse real-world images. Our code, models, and generated 3D assets are available at https://github.com/guochengqian/Magic123.

• The paper introduces a two-stage framework that leverages both 2D and 3D diffusion priors to create high-quality textured 3D meshes from a single image.
• It employs a coarse-to-fine approach using Instant-NGP for rapid NeRF approximation and DMTet for fine mesh refinement.
• Experimental evaluations on synthetic and real-world datasets demonstrate superior performance, paving the way for advanced applications in VR, AR, and digital content creation.

Overview of "Magic123: Redefining Image-to-3D Object Generation"

The paper "Magic123" introduces an innovative methodology for generating high-quality textured 3D meshes from a single unposed image, leveraging both 2D and 3D diffusion priors. This dual-stage technique, comprising of a coarse-to-fine optimization approach, represents a significant enhancement over existing single-image 3D reconstruction methods.

Methodological Insights

Two-Stage Coarse-to-Fine Framework: The proposed framework of Magic123 operates in two distinct phases to achieve its objectives:

1. Coarse Stage: Initially, a neural radiance field (NeRF) is employed to approximate the underlying geometric structure of the scene. Instant-NGP is utilized as the NeRF implementation, favored for its rapid computation and adeptness in handling complex geometries.
2. Fine Stage: Subsequently, the coarse 3D structure undergoes refinement to improve its fidelity and rendering resolution. This involves optimizing a differentiable and memory-efficient mesh representation, specifically using Deep Marching Tetrahedra (DMTet), to produce detailed geometry and texture.

Joint 2D and 3D Diffusion Priors: A notable innovation in this paper is the integration of both 2D and 3D diffusion models to guide the synthesis of novel views. While the 2D priors, driven by score distillation sampling (SDS) with Stable Diffusion, contribute imaginative abilities to explore potential geometries, the 3D priors ensure geometric precision and consistency. The balance between these priors is controlled through a trade-off parameter, enabling the modulation of exploration and exploitation levels during the generation process.

Additional Enhancements: The methodology also incorporates several auxiliary strategies to enhance output quality, including textual inversion for preserving object-specific visual characteristics and monocular depth regularization to prevent degenerate geometrical representations.

Experimental Validation

Magic123 was evaluated on synthetic and real-world datasets, specifically NeRF4 and RealFusion15, showing superior performance across multiple metrics, such as PSNR, LPIPS, and CLIP-similarity. The pipeline's ability to generate coherent and high-resolution 3D models consistently outperformed contemporary methods such as Zero-1-to-3 and RealFusion, as demonstrated through both qualitative and quantitative assessments.

Practical Implications and Future Directions

The proposed framework significantly narrows the gap between machine-driven 3D reconstruction and human-level perceptual capabilities, facilitating applications in areas such as virtual reality, augmented reality, and digital content creation. The research further emphasizes the importance of harmonizing exploratory and exploitative tendencies in generative processes, suggesting avenues for future development in adaptive and context-aware AI models.

Limitations and Challenges: Despite its advancements, the approach does assume a 'front-view' reference image, which can limit performance in certain scenarios. Additionally, reliance on segmentation and monocular depth estimation as preprocessing steps introduces potential dependencies that can affect ultimate output quality.

Overall, Magic123 offers a robust foundation for future exploration in single-image 3D model generation, proposing a versatile framework adaptable to various contexts through the dynamic interplay of 2D and 3D data-driven priors.