Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
175 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Efficient 3D Articulated Human Generation with Layered Surface Volumes (2307.05462v1)

Published 11 Jul 2023 in cs.CV

Abstract: Access to high-quality and diverse 3D articulated digital human assets is crucial in various applications, ranging from virtual reality to social platforms. Generative approaches, such as 3D generative adversarial networks (GANs), are rapidly replacing laborious manual content creation tools. However, existing 3D GAN frameworks typically rely on scene representations that leverage either template meshes, which are fast but offer limited quality, or volumes, which offer high capacity but are slow to render, thereby limiting the 3D fidelity in GAN settings. In this work, we introduce layered surface volumes (LSVs) as a new 3D object representation for articulated digital humans. LSVs represent a human body using multiple textured mesh layers around a conventional template. These layers are rendered using alpha compositing with fast differentiable rasterization, and they can be interpreted as a volumetric representation that allocates its capacity to a manifold of finite thickness around the template. Unlike conventional single-layer templates that struggle with representing fine off-surface details like hair or accessories, our surface volumes naturally capture such details. LSVs can be articulated, and they exhibit exceptional efficiency in GAN settings, where a 2D generator learns to synthesize the RGBA textures for the individual layers. Trained on unstructured, single-view 2D image datasets, our LSV-GAN generates high-quality and view-consistent 3D articulated digital humans without the need for view-inconsistent 2D upsampling networks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. Panohead: Geometry-aware 3d full-head synthesis in 360∘{}^{\circ}start_FLOATSUPERSCRIPT ∘ end_FLOATSUPERSCRIPT. arXiv preprint arXiv:2303.13071, 2023.
  2. 2d human pose estimation: New benchmark and state of the art analysis. In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition, pages 3686–3693, 2014.
  3. ClipFace: Text-guided Editing of Textured 3D Morphable Models. In ArXiv preprint arXiv:2212.01406, 2022.
  4. Gaudi: A neural architect for immersive 3d scene generation. Advances in Neural Information Processing Systems, 35:25102–25116, 2022.
  5. Generative neural articulated radiance fields. In NeurIPS, 2022.
  6. Dreamavatar: Text-and-shape guided 3d human avatar generation via diffusion models, 2023.
  7. pi-GAN: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In CVPR, 2021.
  8. Efficient geometry-aware 3d generative adversarial networks. In CVPR, 2022.
  9. Sofgan: A portrait image generator with dynamic styling. ACM Transactions on Graphics (TOG), 41(1):1–26, 2022.
  10. Gram: Generative radiance manifolds for 3d-aware image generation. In CVPR, 2022.
  11. Unconstrained scene generation with locally conditioned radiance fields. In CVPR, pages 14304–14313, 2021.
  12. Insetgan for full-body image generation. In CVPR, pages 7723–7732, 2022.
  13. Stylegan-human: A data-centric odyssey of human generation. In ECCV, pages 1–19, 2022a.
  14. Stylegan-human: A data-centric odyssey of human generation. In ECCV, 2022b.
  15. 3D shape induction from 2D views of multiple objects. In 3DV, 2017.
  16. Get3d: A generative model of high quality 3d textured shapes learned from images. In NeurIPS, 2022.
  17. Stylepeople: A generative model of fullbody human avatars. In CVPR, pages 5151–5160, 2021.
  18. Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis. In Int. Conf. Learn. Represent., 2022.
  19. GANcraft: Unsupervised 3D neural rendering of minecraft worlds. In ICCV, 2021.
  20. Escaping Plato’s cave: 3D shape from adversarial rendering. In ICCV, 2019.
  21. Avatarclip: Zero-shot text-driven generation and animation of 3d avatars. ACM Transactions on Graphics (TOG), 41(4):1–19, 2022.
  22. EVA3d: Compositional 3d human generation from 2d image collections. In ICLR, 2023.
  23. Humangen: Generating human radiance fields with explicit priors. In CVPR, 2023.
  24. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.
  25. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
  26. Analyzing and improving the image quality of StyleGAN. In CVPR, 2020.
  27. Alias-free generative adversarial networks. In NeurIPS, 2021.
  28. Modular primitives for high-performance differentiable rendering. ACM Transactions on Graphics, 39(6), 2020.
  29. C. Lassner and M. Zollhofer. Pulsar: Efficient sphere-based neural rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1440–1449, 2021.
  30. Pose space deformation: a unified approach to shape interpolation and skeleton-driven deformation. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 165–172, 2000.
  31. Learn to dance with aist++: Music conditioned 3d dance generation, 2021.
  32. Towards unsupervised learning of generative models for 3D controllable image synthesis. In CVPR, 2020.
  33. Magic3d: High-resolution text-to-3d content creation. arXiv preprint arXiv:2211.10440, 2022.
  34. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In CVPR, 2016.
  35. Smpl: A skinned multi-person linear model. ACM transactions on graphics (TOG), 34(6):1–16, 2015.
  36. HoloGAN: Unsupervised learning of 3d representations from natural images. In ICCV, 2019.
  37. BlockGAN: Learning 3D object-aware scene representations from unlabelled images. In NeurIPS, 2020.
  38. M. Niemeyer and A. Geiger. GIRAFFE: Representing scenes as compositional generative neural feature fields. In CVPR, 2021.
  39. Unsupervised learning of efficient geometry-aware neural articulated representations. In ECCV, pages 597–614, 2022.
  40. Stylesdf: High-resolution 3d-consistent image and geometry generation. In CVPR, 2022.
  41. A shading-guided generative implicit model for shape-accurate 3d-aware image synthesis. In NeurIPS, 2021.
  42. Expressive body capture: 3d hands, face, and body from a single image. In CVPR, 2019.
  43. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988, 2022.
  44. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
  45. GRAF: Generative radiance fields for 3d-aware image synthesis. In NeurIPS, 2020.
  46. Voxgraf: Fast 3d-aware image synthesis with sparse voxel grids. arXiv preprint arXiv:2206.07695, 2022.
  47. 3d-aware indoor scene synthesis with depth priors. In ECCV, pages 406–422. Springer, 2022.
  48. 3d neural field generation using triplane diffusion. In CVPR, 2023.
  49. Epigraf: Rethinking training of 3d gans. arXiv preprint arXiv:2206.10535, 2022.
  50. Singraf: Learning a 3d generative radiance field for a single scene. arXiv preprint arXiv:2211.17260, 2022.
  51. Fenerf: Face editing in neural radiance fields. In CVPR, 2022.
  52. Next3d: Generative neural texture rasterization for 3d-aware head avatars. In CVPR, 2023.
  53. Unsupervised generative 3D shape learning from natural images. arXiv preprint arXiv:1910.00287, 2019.
  54. Disentangled3d: Learning a 3d generative model with disentangled geometry and appearance from monocular images. In CVPR, 2022a.
  55. Advances in neural rendering. In Computer Graphics Forum, volume 41, pages 703–735. Wiley Online Library, 2022b.
  56. R. Tucker and N. Snavely. Single-view view synthesis with multiplane images. In CVPR, 2020.
  57. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. arXiv preprint arXiv:2212.00774, 2022.
  58. Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. Advances in neural information processing systems, 29, 2016.
  59. Gram-hd: 3d-consistent image generation at high resolution with generative radiance manifolds. arXiv preprint arXiv:2206.07255, 2022.
  60. Discoscene: Spatially disentangled generative radiance fields for controllable 3d-aware scene synthesis. arXiv preprint arXiv:2212.11984, 2022a.
  61. 3d-aware image synthesis via learning structural and textural representations. In CVPR, 2022b.
  62. Giraffe hd: A high-resolution 3d-aware generative model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18440–18449, 2022.
  63. 3dhumangan: Towards photo-realistic 3d-aware human image generation. arXiv preprint, arXiv:2212.07378, 2022.
  64. 3d-aware semantic-guided generative model for human synthesis. In ECCV, pages 339–356. Springer, 2022a.
  65. Avatargen: A 3d generative model for animatable human avatars. ArXiv, 2023a.
  66. Dreamface: Progressive generation of animatable 3d faces under text guidance. arXiv preprint arXiv:2304.03117, 2023b.
  67. Multi-view consistent generative adversarial networks for 3d-aware image synthesis. In CVPR, pages 18450–18459, 2022b.
  68. Generative multiplane images: Making a 2d gan 3d-aware. In ECCV, pages 18–35. Springer, 2022.
  69. Cips-3d: A 3d-aware generator of gans based on conditionally-independent pixel synthesis. arXiv preprint arXiv:2110.09788, 2021.
  70. Stereo magnification: Learning view synthesis using multiplane images. ACM. Trans. Graph. (SIGGRAPH), 2018.
  71. Visual object networks: image generation with disentangled 3D representations. In NeurIPS, 2018.
Citations (7)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now