Unleashing Vecset Diffusion Model for Fast Shape Generation (2503.16302v2)

Abstract: 3D shape generation has greatly flourished through the development of so-called "native" 3D diffusion, particularly through the Vecset Diffusion Model (VDM). While recent advancements have shown promising results in generating high-resolution 3D shapes, VDM still struggles with high-speed generation. Challenges exist because of difficulties not only in accelerating diffusion sampling but also VAE decoding in VDM, areas under-explored in previous works. To address these challenges, we present FlashVDM, a systematic framework for accelerating both VAE and DiT in VDM. For DiT, FlashVDM enables flexible diffusion sampling with as few as 5 inference steps and comparable quality, which is made possible by stabilizing consistency distillation with our newly introduced Progressive Flow Distillation. For VAE, we introduce a lightning vecset decoder equipped with Adaptive KV Selection, Hierarchical Volume Decoding, and Efficient Network Design. By exploiting the locality of the vecset and the sparsity of shape surface in the volume, our decoder drastically lowers FLOPs, minimizing the overall decoding overhead. We apply FlashVDM to Hunyuan3D-2 to obtain Hunyuan3D-2 Turbo. Through systematic evaluation, we show that our model significantly outperforms existing fast 3D generation methods, achieving comparable performance to the state-of-the-art while reducing inference time by over 45x for reconstruction and 32x for generation. Code and models are available at https://github.com/Tencent/FlashVDM.

Unleashing Vecset Diffusion Model for Fast Shape Generation

The paper "Unleashing Vecset Diffusion Model for Fast Shape Generation" introduces the Flash Vecset Diffusion Model (FlashVDM) as a novel framework designed to expedite the generation of 3D shapes. By addressing the limitations inherent in existing Vecset Diffusion Models (VDMs)—notably their inefficiencies in high-speed generation—this work significantly reduces both VAE decoding times and diffusion sampling steps.

To optimize the Vecset models, FlashVDM focuses on two primary components: diffusion acceleration and VAE acceleration. On the diffusion front, the authors identify the instability issues prevalent in 3D distillation methods when applied directly. In response, they propose Progressive Flow Distillation, a multi-phase method incorporating consistency flow distillation and adversarial fine-tuning. This enables comparable shape generation using only 5 steps, significantly lowering the Number of Function Evaluations (NFE) compared to previous models.

On the VAE decoding side, FlashVDM introduces innovative techniques such as Adaptive KV Selection, Hierarchical Volume Decoding, and an Efficient Decoder Architecture. The hierarchical approach acknowledges the sparsity of 3D shape surfaces and focuses computational efforts effectively where needed, reducing FLOPs by 97.1%. Furthermore, adaptive KV selection exploits locality within shape queries to further minimize the computational burden, while the newly proposed efficient decoder architecture trims down the overhead through refined network design choices.

The results presented in the paper showcase substantial improvements over existing fast 3D generation methods. FlashVDM achieves a reduction in inference time by over 45 times for reconstruction and 32 times for generation, without significantly compromising quality—demonstrating comparable performance to state-of-the-art methods. These advancements highlight the potential of FlashVDM in real-time, high-fidelity 3D applications, suggesting future developments in interactive and dynamic AI-driven 3D modeling environments.

The implications of this research are twofold. Practically, FlashVDM positions itself as a key player in areas demanding rapid yet accurate 3D shape generation, such as virtual reality, gaming, and simulation. Theoretically, this work urges a reconsideration of current diffusion models, advocating for methods that harmonize speed and precision—a crucial balance for advancing AI's capabilities in 3D space.

Moving forward, there remains the potential to explore further reductions in diffusion sampling, possibly through one-step distillation techniques. Additionally, the interaction between real-world data and reinforcement strategies in enhancing model robustness and fidelity presents a promising avenue for future exploration. As such, FlashVDM not only represents a significant stride towards efficient 3D generation but also lays down a compelling groundwork for subsequent innovation in AI-driven modeling and simulation.