OctFusion: Octree-based Diffusion Models for 3D Shape Generation (2408.14732v2)

Abstract: Diffusion models have emerged as a popular method for 3D generation. However, it is still challenging for diffusion models to efficiently generate diverse and high-quality 3D shapes. In this paper, we introduce OctFusion, which can generate 3D shapes with arbitrary resolutions in 2.5 seconds on a single Nvidia 4090 GPU, and the extracted meshes are guaranteed to be continuous and manifold. The key components of OctFusion are the octree-based latent representation and the accompanying diffusion models. The representation combines the benefits of both implicit neural representations and explicit spatial octrees and is learned with an octree-based variational autoencoder. The proposed diffusion model is a unified multi-scale U-Net that enables weights and computation sharing across different octree levels and avoids the complexity of widely used cascaded diffusion schemes. We verify the effectiveness of OctFusion on the ShapeNet and Objaverse datasets and achieve state-of-the-art performances on shape generation tasks. We demonstrate that OctFusion is extendable and flexible by generating high-quality color fields for textured mesh generation and high-quality 3D shapes conditioned on text prompts, sketches, or category labels. Our code and pre-trained models are available at https://github.com/octree-nn/octfusion.

• The paper introduces OctFusion, advancing 3D shape generation by integrating octree-based latent representations with a unified multi-scale diffusion model.
• It employs a U-Net architecture with shared weights across octree levels and local SDF decoding via MLPs to reduce complexity and speed up mesh conversion.
• Empirical results on ShapeNet and Objaverse demonstrate its rapid, high-resolution mesh creation and robust performance in both unconditional and conditional scenarios.

OctFusion: Octree-based Diffusion Models for 3D Shape Generation

In the domain of 3D shape generation, the need for efficient and high-quality methods has become increasingly critical due to applications spanning from virtual reality to gaming. The paper "OctFusion: Octree-based Diffusion Models for 3D Shape Generation" addresses existing challenges in generating diverse and high-resolution 3D shapes through diffusion models. The authors present a novel framework, OctFusion, utilizing octree-based latent representations coupled with a unified multi-scale diffusion model.

OctFusion advances the capability to generate 3D shapes with resolutions initially unattained by existing methodologies, due primarily to its efficient representation of 3D data and the sharing mechanism within its diffusion model. The key innovation lies in employing a volumetric octree structure augmented by latent features at each node, which are decoded into local signed distance fields (SDFs) using shared multilayer perceptrons (MLPs). This combination exploits the strengths of both implicit and explicit representations, thereby supporting the creation of detailed and continuous shape surfaces with resolutions up to 1024.

The unified diffusion model proposed in this paper diverges from traditional cascaded approaches by enabling weight and computation sharing across various octree levels. The significance of this lies in reducing training complexity and enabling efficient and fine-grained shape generation. Specifically, the model leverages a U-Net architecture designed to revert the noising process intrinsic to diffusion models, effectively predicting clean signals from noised octrees. This strategy, coupled with octree hierarchical structure considerations, allows OctFusion to generate detailed shapes efficiently even at higher resolutions.

Empirically, OctFusion was assessed against several prominent datasets, showcasing superior performance in both unconditional and conditional scenarios. Numerical results highlight its proficiency with high retrieval scores on the ShapeNet and Objaverse datasets. Specifically, the model exhibited a capacity to convert implicit fields to meshes in under 2.5 seconds on an Nvidia 4090 GPU, outperforming many contemporary frameworks.

Additionally, OctFusion's framework demonstrates significant generalization capabilities. It supports text/sketch-conditioned generation and textured mesh generation, broadening its adaptability and application scope. By extending the diffusion model to color fields, OctFusion adds another dimension to its 3D modeling potential, establishing itself as a versatile tool in conditional generation tasks.

The theoretical implications of OctFusion's development lie in its novel combination of octree-based latent spaces and unified diffusion processing, signaling improved pathways in achieving high-resolution 3D shape modeling. This work also potentially reduces the burden of computational requirements typically associated with high-resolution 3D generation, promoting practical applications within resource-constrained environments.

Speculating on future developments, OctFusion could be further optimized and scaled to accommodate even more complex datasets and larger models. Possible directions for exploration might include the integration of more sophisticated latent space manipulation techniques or the expansion into real-time generation systems.

Overall, OctFusion contributes a substantial advancement in the field of 3D shape generation, providing a robust and efficient framework that effectively balances quality with computational feasibility. Its adoption could influence forthcoming methodologies, setting a precedence for models that aim to harness the combined power of latent and explicit geometric representations in conjunction with novel diffusion techniques.