Structured 3D Latents for Scalable and Versatile 3D Generation (2412.01506v3)

Abstract: We introduce a novel 3D generation method for versatile and high-quality 3D asset creation. The cornerstone is a unified Structured LATent (SLAT) representation which allows decoding to different output formats, such as Radiance Fields, 3D Gaussians, and meshes. This is achieved by integrating a sparsely-populated 3D grid with dense multiview visual features extracted from a powerful vision foundation model, comprehensively capturing both structural (geometry) and textural (appearance) information while maintaining flexibility during decoding. We employ rectified flow transformers tailored for SLAT as our 3D generation models and train models with up to 2 billion parameters on a large 3D asset dataset of 500K diverse objects. Our model generates high-quality results with text or image conditions, significantly surpassing existing methods, including recent ones at similar scales. We showcase flexible output format selection and local 3D editing capabilities which were not offered by previous models. Code, model, and data will be released.

• The paper presents a unified Structured Latent framework that fuses multiview image features into a sparse 3D grid for versatile asset generation.
• It uses a two-stage pipeline and rectified flow transformers trained on up to 500,000 3D objects to achieve significant improvements in PSNR, LPIPS, and CLIP metrics.
• The method supports flexible decoding into diverse formats, including radiance fields, 3D Gaussians, and meshes, advancing applications in gaming, film, and virtual reality.

A Novel Approach to Scalable and Versatile 3D Generation

Introduction

The paper "Structured 3D Latents for Scalable and Versatile 3D Generation" presents a novel method for generating high-quality and versatile 3D assets. The core innovation of this research is the introduction of a unified Structured Latent (SL) AT representation, which captures both geometric and textural information for flexible decoding into various 3D formats. This novel representation supports high-fidelity and efficient 3D asset creation, demonstrating significant improvements over existing methods.

Overview of Methodology

The cornerstone of this research is the SL AT representation, which fuses localized image features from multiview inputs into a sparse 3D grid. This representation allows for decoding into multiple output formats, including Radiance Fields, 3D Gaussians, and meshes. The SL AT approach addresses the current gap between high-quality 2D image generation and the relatively underdeveloped 3D asset generation by creating a unified latent space that accommodates diverse downstream requirements.

The framework deploys rectified flow transformers trained on extensive datasets comprising up to 500,000 diverse 3D objects. These models, dubbed TRELLIS in the paper, leverage text and image conditions for asset generation. The process is divided into a two-stage pipeline: the initial stage generates the sparse structure, followed by the generation of local latent vectors. This modular approach supports versatile output format selection and flexible local 3D editing.

Numerical Results and Comparative Evaluation

The paper reports strong numerical results indicating the superiority of this approach. The reconstructed assets significantly outperform previous methods in terms of appearance fidelity and geometric accuracy, as demonstrated by higher PSNR and lower LPIPS scores on a comprehensive evaluation dataset (Toys4k). The improvement in geometry is particularly notable, exceeding methodologies like CLAY focusing solely on shape encoding.

In quantitative comparisons, the TRELLIS model showed an improved CLIP score and reduced Frechet Distance metrics when benchmarking against other state-of-the-art 3D generation methods. This underscores its enhanced ability to generate visually appealing and prompt-aligned 3D assets.

Qualitative comparisons further reinforce these findings, with TRELLIS-generated assets displaying more intricate detailing and realistic rendering across both text-to-3D and image-to-3D tasks. Contrasting results from alternative models often exhibited featureless appearances or geometric inaccuracies, highlighting the robustness of the SL AT framework.

Practical and Theoretical Implications

This research significantly advances the field of 3D generative modeling, establishing a scalable methodology that bridges the gap between diverse 3D representations. The incorporation of a powerful vision foundation model allows for high-resolution encoding without pre-fitting 3D objects, streamlining the model training process. The versatility in output formats and the ability for detailed local editing positions this model as a promising tool for applications in gaming, film, and virtual reality production.

Future Directions

The introduction of the SL AT framework opens numerous avenues for continued exploration. Future research may extend the flexibility of the framework by integrating other advanced vision models or exploring alternative sparse structure representations. Additionally, the efficiency of the two-stage generation process might be further optimized, potentially leveraging innovations in parallel processing or attention mechanisms.

Conclusion

The paper introduces a pivotal development in the field of 3D asset generation, offering a robust methodology for creating high-quality, versatile, and editable 3D graphics. The SL AT representation, alongside the TRELLIS model trained on expansive datasets, marks a substantial leap forward, potentially serving as a foundation for future explorations in 3D generative models. The work sets a new standard for what is achievable and encourages further innovation towards fully integrated 3D asset pipelines in digital content creation.