Locally Attentional SDF Diffusion for Controllable 3D Shape Generation (2305.04461v2)

Abstract: Although the recent rapid evolution of 3D generative neural networks greatly improves 3D shape generation, it is still not convenient for ordinary users to create 3D shapes and control the local geometry of generated shapes. To address these challenges, we propose a diffusion-based 3D generation framework -- locally attentional SDF diffusion, to model plausible 3D shapes, via 2D sketch image input. Our method is built on a two-stage diffusion model. The first stage, named occupancy-diffusion, aims to generate a low-resolution occupancy field to approximate the shape shell. The second stage, named SDF-diffusion, synthesizes a high-resolution signed distance field within the occupied voxels determined by the first stage to extract fine geometry. Our model is empowered by a novel view-aware local attention mechanism for image-conditioned shape generation, which takes advantage of 2D image patch features to guide 3D voxel feature learning, greatly improving local controllability and model generalizability. Through extensive experiments in sketch-conditioned and category-conditioned 3D shape generation tasks, we validate and demonstrate the ability of our method to provide plausible and diverse 3D shapes, as well as its superior controllability and generalizability over existing work. Our code and trained models are available at https://zhengxinyang.github.io/projects/LAS-Diffusion.html

• The paper introduces a novel two-stage diffusion model that converts noise into coarse 3D shapes and refines them into detailed forms using SDF diffusion.
• It employs a view-aware local attention mechanism to harness 2D sketch features for precise control over 3D shape generation.
• The approach enables intuitive, rapid prototyping and broadens creative possibilities in 3D modeling with impressive generality.

Expanding the Horizons of 3D Shape Generation with LAS-Diffusion

Overview

In the rapidly evolving field of 3D shape generation, the paper on "Locally Attentional SDF Diffusion for Controllable 3D Shape Generation" introduces an innovative approach that bridges the gap between user intention and the automated creation of complex 3D shapes. This research developed by Xin-Yang Zheng et al. from a collaboration between Tsinghua University, Peking University, and Microsoft Research Asia, leverages a diffusion-based framework termed locally attentional SDF (Signed Distance Function) diffusion, or LAS-Diffusion, for generating diverse and high-quality 3D shapes from simple 2D sketches.

Technical Approach

The LAS-Diffusion model encapsulates a two-stage diffusion process designed for efficient 3D shape synthesis. The initial stage involves an 'occupancy-diffusion' which transforms noise into a coarse representation of the target shape, laying the groundwork for the structure. The second 'SDF-diffusion' stage refines this structure into a high-resolution SDF, capturing the intricate details of the 3D shape. The pivotal innovation within this framework is its unique view-aware local attention mechanism. This mechanism allows the model to use local features extracted from 2D image sketches for guiding the shape generation process, enabling remarkable control over the final 3D output.

Methodological Insights

• Two-Stage Diffusion: The two-staged approach efficiently manages high-resolution 3D data, making the model both practical and scalable.
• Local Attention Mechanism: By leveraging local image features, the model achieves an unprecedented level of controllability and fidelity in synthesizing 3D shapes that align with the user's conceptual sketches.
• Generative Capabilities: The experiments demonstrate the model's robustness and adaptability across various conditions, including the generation of novel shapes not present in the training data, showcasing superior generality and creativity.

Practical and Theoretical Implications

From a practical standpoint, this research opens new avenues for intuitive 3D modeling, significantly lowering the barrier for non-experts to bring their imaginative concepts to life. In professional settings, it can streamline the design process, offering a rapid prototyping tool that responds accurately to sketch-based inputs.

Theoretically, the paper contributes to understanding the intersection between local feature attention mechanisms and generative modeling of complex structures. It further illuminates the path for future research on conditional 3D generation, particularly in leveraging mixed-modal inputs for more comprehensive and user-intuitive generative processes.

Speculations on Future Developments

Looking ahead, the introduced LAS-Diffusion framework suggests several exciting directions for further investigation and development. The integration of additional input modalities, such as textual descriptions alongside sketches, could enrich the model's understanding and generative capabilities. Additionally, exploring multi-view or sequential sketch inputs may provide deeper insights into capturing and rendering the envisioned 3D shapes with even greater accuracy.

Conclusion

In summary, "Locally Attentional SDF Diffusion for Controllable 3D Shape Generation" by Xin-Yang Zheng and colleagues marks a significant step forward in the domain of 3D shape generation. By effectively combining local attention mechanisms with a novel two-stage diffusion process, the research not only achieves high fidelity in 3D shape synthesis but also remarkably enhances user controllability. This work not only contributes valuable insights to the academic community but also holds promising potential for various practical applications in design and digital content creation.