GaussianDreamerPro: Text to Manipulable 3D Gaussians with Highly Enhanced Quality (2406.18462v1)

Abstract: Recently, 3D Gaussian splatting (3D-GS) has achieved great success in reconstructing and rendering real-world scenes. To transfer the high rendering quality to generation tasks, a series of research works attempt to generate 3D-Gaussian assets from text. However, the generated assets have not achieved the same quality as those in reconstruction tasks. We observe that Gaussians tend to grow without control as the generation process may cause indeterminacy. Aiming at highly enhancing the generation quality, we propose a novel framework named GaussianDreamerPro. The main idea is to bind Gaussians to reasonable geometry, which evolves over the whole generation process. Along different stages of our framework, both the geometry and appearance can be enriched progressively. The final output asset is constructed with 3D Gaussians bound to mesh, which shows significantly enhanced details and quality compared with previous methods. Notably, the generated asset can also be seamlessly integrated into downstream manipulation pipelines, e.g. animation, composition, and simulation etc., greatly promoting its potential in wide applications. Demos are available at https://taoranyi.com/gaussiandreamerpro/.

GaussianDreamerPro: Text to Manipulable 3D Gaussians with Highly Enhanced Quality

The paper "GaussianDreamerPro: Text to Manipulable 3D Gaussians with Highly Enhanced Quality" introduces GaussianDreamerPro, an innovative framework aimed at significantly enhancing the quality of text-to-3D Gaussian asset generation. This work builds on the emerging success of 3D Gaussian splatting (3D-GS) in 3D reconstruction and rendering, and seeks to bridge the gap between high-quality rendering in reconstruction tasks and generation tasks.

Background and Motivation

3D Gaussian splatting has demonstrated notable efficacy in rendering realistic 3D scenes quickly. However, extending these benefits to text-to-3D generation has been challenging. Previous methods have struggled to achieve the same level of detail and quality in generated 3D assets as seen in reconstruction tasks, primarily due to uncontrolled Gaussian growth producing indeterminate and blurred assets.

Contributions

The authors propose GaussianDreamerPro as a solution, with the central idea of binding Gaussians to dynamically evolving geometry throughout the generation process. This approach is designed to progressively enrich both geometry and appearance, yielding assets with improved quality and significantly enhanced details. The framework consists of two main stages: basic 3D asset generation and quality enhancement with geometry-bound Gaussians.

Methodology

Basic 3D Asset Generation

Firstly, a coarse 3D asset is generated using a 3D diffusion model, which provides initial geometry guidance. This is followed by transformation into 2D Gaussians optimized using a 2D diffusion model. This two-step process leverages the distinct strengths of both 3D and 2D diffusion models, ultimately producing a basic 3D asset with reasonable geometry and appearance.

Quality Enhancement

The subsequent quality enhancement stage involves constructing 3D Gaussians bound to a mesh derived from the basic 3D asset. This binding constrains Gaussian growth, allowing for controlled, progressive optimization of both geometry and appearance. The enhanced 3D assets are three-dimensionally consistent and exhibit fine details, which overcome the limitations of previous methods where free-form Gaussian splatting led to instability and blurriness.

Key Results and Comparisons

GaussianDreamerPro demonstrates superior performance when compared to existing methods such as LucidDreamer, DreamCraft3D, DreamFusion, Magic3D, Fantasia3D, GaussianDreamer, and GSGEN. Visual comparisons presented in the paper show that GaussianDreamerPro achieves higher clarity, better geometry consistency, and overall superior quality in rendered assets. Additionally, user studies indicate a clear preference for assets generated by GaussianDreamerPro, affirming its practical advantages.

Implications and Future Directions

The introduction of geometry-bound Gaussians for text-to-3D generation has significant implications. By constraining Gaussian growth, the method ensures detailed and stable asset generation, making it more suitable for practical applications in gaming, movies, and extended reality (XR). The compatibility with other 3D generation frameworks, exemplified by the successful enhancement of assets generated by DreamCraft3D, suggests a broad utility and potential for integrating GaussianDreamerPro with various 3D asset creation pipelines.

Future developments might focus on addressing the method's limitations in handling complex scenes involving multiple objects. Enhancements to the guiding diffusion models and pretraining on datasets encompassing multiple objects may offer solutions, paving the way for even more versatile and high-quality 3D asset generation.

Conclusion

GaussianDreamerPro marks a significant step forward in the field of text-to-3D asset generation, leveraging the strengths of 3D Gaussian splatting combined with geometry constraints to deliver high-quality, manipulable 3D assets. This work opens promising avenues for practical applications and sets a solid foundation for future research and development in AI-driven 3D content creation.