BrightDreamer: Generic 3D Gaussian Generative Framework for Fast Text-to-3D Synthesis (2403.11273v2)

Abstract: Text-to-3D synthesis has recently seen intriguing advances by combining the text-to-image priors with 3D representation methods, e.g., 3D Gaussian Splatting (3D GS), via Score Distillation Sampling (SDS). However, a hurdle of existing methods is the low efficiency, per-prompt optimization for a single 3D object. Therefore, it is imperative for a paradigm shift from per-prompt optimization to feed-forward generation for any unseen text prompts, which yet remains challenging. An obstacle is how to directly generate a set of millions of 3D Gaussians to represent a 3D object. This paper presents BrightDreamer, an end-to-end feed-forward approach that can achieve generalizable and fast (77 ms) text-to-3D generation. Our key idea is to formulate the generation process as estimating the 3D deformation from an anchor shape with predefined positions. For this, we first propose a Text-guided Shape Deformation (TSD) network to predict the deformed shape and its new positions, used as the centers (one attribute) of 3D Gaussians. To estimate the other four attributes (i.e., scaling, rotation, opacity, and SH), we then design a novel Text-guided Triplane Generator (TTG) to generate a triplane representation for a 3D object. The center of each Gaussian enables us to transform the spatial feature into the four attributes. The generated 3D Gaussians can be finally rendered at 705 frames per second. Extensive experiments demonstrate the superiority of our method over existing methods. Also, BrightDreamer possesses a strong semantic understanding capability even for complex text prompts. The code is available in the project page.

BrightDreamer: Pioneering Fast Text-to-3D Generation via 3D Gaussian Generative Framework

Introduction

The recent endeavor in synthesizing 3D content from textual descriptions has garnered substantial interest, particularly for its myriad applications ranging from game development to virtual reality. Pioneering methods integrating text-to-image models with 3D representations, particularly Gaussian Splatting (GS), have marked significant advancements. Nevertheless, these methodologies are encumbered by inefficiencies, predominantly due to their reliance on iterative, per-prompt optimizations for single 3D object creation. BrightDreamer emerges as a novel framework designed to revolutionize this space by providing a generic, single-stage approach for rapid text-to-3D synthesis.

Methodology

BrightDreamer’s architecture innovatively conceptualizes the 3D object generation process. This generative framework leverages an end-to-end approach to produce 3D content from textual prompts quickly. At its core, the framework aims to generate a set of millions of 3D Gaussians to represent objects directly, effectively bypassing the iterative optimization processes that plague existing methods. The strategy includes:

• Text-guided Shape Deformation (TSD) for predicting the deformed shape from an anchor shape.
• A novel Text-guided Triplane Generator (TTG) for generating spatial representation through triplanes.
• A sophisticated Gaussian Decoder for deducing 3D Gaussian attributes from spatial features.

Through spatial transformations and a detailed rendering pipeline that includes both shape deformation and attribute generation, BrightDreamer significantly reduces generation latency to an unprecedented 77 ms, with rendering capabilities reaching 705 frames per second for the generated content.

Experimental Results

The comparative analysis and experimental validation underscore BrightDreamer's superiority over existing techniques, reflecting both in terms of generation speed and semantic comprehension of complex textual prompts. The framework demonstrates a high degree of generalization, enabling it to process and accurately render 3D content for prompts never encountered during training. Moreover, BrightDreamer introduces a substantial improvement in rendering speeds and generation latency, establishing new benchmarks for text-to-3D synthesis.

Implications and Future Directions

The introduction of BrightDreamer signals a significant shift in the development of generative models for 3D content, particularly emphasizing the model's adeptness at handling complex, unseen text prompts and its exceptional efficiency. The ability to interpolate between inputs for generating nuanced content further suggests its potential for fostering creativity and expansive exploration in 3D design and virtual content creation.

While BrightDreamer represents a leap towards resolving the inefficiencies of text-to-3D generation, several avenues remain open for future research. The exploration into improving the diversity and variability of generated outcomes from single text prompts presents an interesting challenge. Moreover, expanding the model to accommodate a wider range of textual descriptions and refining its understanding of spatial and relational nuances in text descriptions could further enhance its applicability and accuracy.

Conclusion

BrightDreamer establishes a new paradigm in text-to-3D generation, offering a fast, generalizable, and highly efficient framework capable of synthesizing 3D content from textual prompts. Its introduction not only addresses existing limitations in the field but also opens new pathways for exploration in generative AI, signifying a substantial step forward in the creation of immersive, text-driven 3D environments.