GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models (2310.08529v3)

Abstract: In recent times, the generation of 3D assets from text prompts has shown impressive results. Both 2D and 3D diffusion models can help generate decent 3D objects based on prompts. 3D diffusion models have good 3D consistency, but their quality and generalization are limited as trainable 3D data is expensive and hard to obtain. 2D diffusion models enjoy strong abilities of generalization and fine generation, but 3D consistency is hard to guarantee. This paper attempts to bridge the power from the two types of diffusion models via the recent explicit and efficient 3D Gaussian splatting representation. A fast 3D object generation framework, named as GaussianDreamer, is proposed, where the 3D diffusion model provides priors for initialization and the 2D diffusion model enriches the geometry and appearance. Operations of noisy point growing and color perturbation are introduced to enhance the initialized Gaussians. Our GaussianDreamer can generate a high-quality 3D instance or 3D avatar within 15 minutes on one GPU, much faster than previous methods, while the generated instances can be directly rendered in real time. Demos and code are available at https://taoranyi.com/gaussiandreamer/.

Overview of GaussianDreamer

The paper introduces GaussianDreamer, a novel framework designed to efficiently produce high-quality 3D assets from textual prompts. The process marries the strengths of 3D and 2D diffusion models, utilizing a recent efficient representation known as 3D Gaussian Splatting. This innovative approach enables the rapid generation of 3D objects with rich details and consistency while offering the capability of real-time rendering.

Bridging 2D and 3D Diffusion Models

To capitalize on the distinct advantages of 2D and 3D diffusion models — the former's detail richness and the latter's three-dimensional consistency — GaussianDreamer employs a recent advancement called 3D Gaussian Splatting. This method uses 3D diffusion models to provide a basic geometric form as a starting point, and then enriches that form with details via a 2D diffusion model. The integration of both models mitigates the limitations of working exclusively in either dimension and accelerates the training process considerably as compared to techniques that use 3D training data alone.

Methodology

GaussianDreamer operates in two major steps:

1. An initial 3D object is generated using a 3D diffusion model based on textual prompts, which yields a primitive but coherent structure.
2. This structure is then refined through a 2D diffusion model that optimizes the details of the object's geometry and appearance.

Additional operations such as noisy point growing and color perturbation are applied to enhance the initial geometric structure. The process is notably swift, allowing for the completion of the training within 15 minutes on a single GPU.

Advancements and Applications

The achievement of merging 3D and 2D diffusion model capabilities is significant. Not only does this lead to faster generation times and high-quality output, but it also allows for real-time rendering which is a considerable step forward in the field. The method has practical implications in various industries such as gaming, virtual reality, and film, where speed and quality of 3D asset generation are crucial.

Moreover, the authors claim that the method can be adapted to a wide array of prompts, showing versatility and the potential to generate a broad range of detailed 3D models. Consequently, GaussianDreamer stands out as a user-friendly and powerful tool for rapid 3D content creation.

In summary, GaussianDreamer showcases a leap in 3D asset generation technology by swiftly producing realistic 3D models that blend coherence and detail, ultimately benefiting various industries reliant on 3D content.