DSG-Net: Learning Disentangled Structure and Geometry for 3D Shape Generation (2008.05440v4)

Abstract: D shape generation is a fundamental operation in computer graphics. While significant progress has been made, especially with recent deep generative models, it remains a challenge to synthesize high-quality shapes with rich geometric details and complex structure, in a controllable manner. To tackle this, we introduce DSG-Net, a deep neural network that learns a disentangled structured and geometric mesh representation for 3D shapes, where two key aspects of shapes, geometry, and structure, are encoded in a synergistic manner to ensure plausibility of the generated shapes, while also being disentangled as much as possible. This supports a range of novel shape generation applications with disentangled control, such as interpolation of structure (geometry) while keeping geometry (structure) unchanged. To achieve this, we simultaneously learn structure and geometry through variational autoencoders (VAEs) in a hierarchical manner for both, with bijective mappings at each level. In this manner, we effectively encode geometry and structure in separate latent spaces, while ensuring their compatibility: the structure is used to guide the geometry and vice versa. At the leaf level, the part geometry is represented using a conditional part VAE, to encode high-quality geometric details, guided by the structure context as the condition. Our method not only supports controllable generation applications but also produces high-quality synthesized shapes, outperforming state-of-the-art methods. The code has been released at https://github.com/IGLICT/DSG-Net.

• The paper introduces a novel representation that decouples 3D shape structure and geometry using hierarchical variational autoencoders.
• The paper leverages a conditional part geometry VAE to capture fine details, achieving superior performance on metrics like Chamfer Distance.
• The approach enables innovative 3D shape interpolation and controlled synthesis, advancing applications in computer graphics and vision.

DSG-Net: Learning Disentangled Structure and Geometry for 3D Shape Generation

The paper introduces DSG-Net, a deep generative network designed to advance the field of 3D shape generation by disentangling structure and geometry into separate latent representations. The core challenge addressed by the paper is synthesizing high-quality 3D shapes with intricate geometric details and complex structures under controlled conditions, a necessity for diverse applications within computer graphics and vision. The paper is contributed by a team from prominent institutions including the Institute of Computing Technology, CAS, University of Chinese Academy of Sciences, Stanford University, and Cardiff University.

Key Contributions

1. Novel Representation: DSG-Net proposes a representation that encodes 3D shapes by decoupling geometry (fine details of parts) and structure (part hierarchy and relationships). This approach is implemented using variational autoencoders (VAEs) in a hierarchical manner, ensuring compatibility via bidirectional mappings between the two aspects, while aiming for maximal disentanglement.
2. Conditional Part Geometry VAE: Critical to the geometry representation, DSG-Net uses a mesh-based approach with a conditional part geometry VAE to capture fine geometric details. This enables specialized part geometry generation contextualized within the overall structure.
3. Disentangled Control in Applications: The disentangled representation permits innovative applications in 3D shape processing. Notably, it supports interpolation where either the geometry or the structure of shapes is altered while keeping the other constant. This allows mixing and matching features from different shapes, offering enhanced control compared to traditional approaches.
4. Evaluation and Results: The paper provides extensive qualitative and quantitative evaluations on multiple shape categories from the PartNet dataset. DSG-Net consistently outperforms state-of-the-art methods, both in terms of geometry and structure representation, as measured by metrics such as Chamfer Distance and HierInsSeg.

Strong Numerical Results and Claims

The paper presents strong claims regarding the superiority of DSG-Net over traditional methods such as StructureNet and SDM-Net, highlighting improved metrics like Chamfer Distance and Earth Mover's Distance. Extensive ablation studies underscore the importance of novel components like the cyclic disentanglement mechanism in enhancing disentanglement and representation effectiveness.

Implications for Theory and Practice

The theoretical implications of DSG-Net are profound, suggesting new paradigms in shape generation that facilitate more granular control over 3D modeling and synthesis tasks. Practically, DSG-Net's disentangled approach may pave the way for advanced applications in automated design, virtual reality, and interactive graphics, where users require precise control over object features.

Future Developments

The research opens avenues for future work in unsupervised learning settings, particularly in disentanglement of complex structural hierarchies without pre-defined part annotations. Moreover, the disentangled representation could be incorporated into hybrid models, integrating texture or materials, further broadening DSG-Net's applicability.

Overall, DSG-Net signifies an advancement in the generative modeling of 3D shapes, extending possibilities for creative and controlled 3D object synthesis in the field of artificial intelligence and computer graphics.