GEM3D: GEnerative Medial Abstractions for 3D Shape Synthesis (2402.16994v2)
Abstract: We introduce GEM3D -- a new deep, topology-aware generative model of 3D shapes. The key ingredient of our method is a neural skeleton-based representation encoding information on both shape topology and geometry. Through a denoising diffusion probabilistic model, our method first generates skeleton-based representations following the Medial Axis Transform (MAT), then generates surfaces through a skeleton-driven neural implicit formulation. The neural implicit takes into account the topological and geometric information stored in the generated skeleton representations to yield surfaces that are more topologically and geometrically accurate compared to previous neural field formulations. We discuss applications of our method in shape synthesis and point cloud reconstruction tasks, and evaluate our method both qualitatively and quantitatively. We demonstrate significantly more faithful surface reconstruction and diverse shape generation results compared to the state-of-the-art, also involving challenging scenarios of reconstructing and synthesizing structurally complex, high-genus shape surfaces from Thingi10K and ShapeNet.
- Learning representations and generative models for 3D point clouds. In Proc. ICML.
- Learning Representations and Generative Models for 3D Point Clouds.
- Jules Bloomenthal and Ken Shoemake. 1991. Convolution surfaces. Proc. ACM SIGGRAPH, 251–256.
- Shapenet: An information-rich 3D model repository. arXiv preprint arXiv:1512.03012 (2015).
- Learning Generative Models of 3D Structures. In Computer Graphics Forum.
- Zhiqin Chen and Hao Zhang. 2019. Learning Implicit Fields for Generative Shape Modeling. CVPR (2019).
- Mattéo Clémot and Julie Digne. 2023. Neural skeleton: Implicit neural representation away from the surface. Computers & Graphics 114 (2023), 368–378.
- Angela Dai and Matthias Nießner. 2019. Scan2Mesh: From Unstructured Range Scans to 3D Meshes. In Proc. CVPR.
- Characterizing structural relationships in scenes using graph kernels. In ACM Transactions on Graphics, Vol. 30. ACM, 34.
- Medial Skeletal Diagram: A Generalized Medial Axis Approach for Compact 3D Shape Representation. arXiv:2310.09395
- Denoising Diffusion Probabilistic Models. In Proc. NIPS.
- A Probabilistic Model for Component-Based Shape Synthesis. In Proc. ACM SIGGRAPH.
- Elucidating the Design Space of Diffusion-Based Generative Models. In Proc. NIPS.
- Diederik P. Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. In Proc. ICLR.
- GRASS: Generative Recursive Autoencoders for Shape Structures. Trans. Graphics 36, 4 (2017).
- Q-MAT: Computing Medial Axis Transform By Quadratic Error Minimization. ACM Transactions on Graphics 35, 1 (2016).
- Magic3D: High-Resolution Text-to-3D Content Creation. In Proc. CVPR.
- Interactive 3D modeling with a generative adversarial network. In Proc. 3DV.
- William E Lorensen and Harvey E Cline. 1987. Marching cubes: A high resolution 3D surface construction algorithm. ACM Transactions on Graphics 21, 4 (1987).
- Computer-Generated Residential Building Layouts. In ACM Transactions on Graphics.
- Occupancy networks: Learning 3D reconstruction in function space. In Proc. CVPR.
- A Survey of Urban Reconstruction. In Computer Graphics Forum.
- Skeleton-bridged point completion: from global inference to local adjustment. In Proc. NIPS.
- Deepsdf: Learning continuous signed distance functions for shape representation. In Proc. CVPR.
- Advances in Data-Driven Analysis and Synthesis of 3D Indoor Scenes. In Computer Graphics Forum.
- Convolutional occupancy networks. In Proc. ECCV.
- Pixologic. 2022. ZBrush. https://www.maxon.net/en/zbrush
- DreamFusion: Text-to-3D using 2D Diffusion. ICLR (2023).
- Przemyslaw Prusinkiewicz and Aristid Lindenmayer. 1990. The Algorithmic Beauty of Plants.
- Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Proc. NIPS.
- Assessing Generative Models via Precision and Recall. In Advances in Neural Information Processing Systems (NeurIPS).
- Deep Generative Models on 3D Representations: A Survey. arXiv:2210.15663 [cs.CV]
- Implicit Neural Representations with Periodic Activation Functions. In Proc. NIPS.
- Score-Based Generative Modeling through Stochastic Differential Equations. In Proc. ICLR.
- Mean Curvature Skeletons. 31, 5 (2012).
- 3D Skeletons: A State-of-the-Art Report. Computer Graphics Forum 35, 2 (2016).
- Curve skeleton extraction from incomplete point cloud. ACM Transactions on Graphics 28 (2009).
- Sphere-Meshes: shape approximation using spherical quadric error metrics. ACM Transactions on Graphics 32, 6 (2013).
- Attention is all you need. In Proc. NIPS.
- Deep points consolidation. ACM Transactions on Graphics 34 (2015).
- A survey of deep learning-based 3D shape generation. Computational Visual Media 9 (2023), 407–442. Issue 3.
- P2P-NET: Bidirectional Point Displacement Net for Shape Transform. ACM Transactions on Graphics 37, 4 (2018).
- Point-bert: Pre-training 3D point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 19313–19322.
- SCALe-invariant Integral Surfaces. Computer Graphics Forum (2013).
- 3DILG: Irregular Latent Grids for 3D Generative Modeling. In Proc. NIPS.
- 3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models. ACM Transactions on Graphics 42, 4 (2023).
- Qingnan Zhou and Alec Jacobson. 2016. Thingi10K: A Dataset of 10,000 3D-Printing Models. arXiv preprint arXiv:1605.04797 (2016).