Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Portrait3D: Text-Guided High-Quality 3D Portrait Generation Using Pyramid Representation and GANs Prior (2404.10394v1)

Published 16 Apr 2024 in cs.CV

Abstract: Existing neural rendering-based text-to-3D-portrait generation methods typically make use of human geometry prior and diffusion models to obtain guidance. However, relying solely on geometry information introduces issues such as the Janus problem, over-saturation, and over-smoothing. We present Portrait3D, a novel neural rendering-based framework with a novel joint geometry-appearance prior to achieve text-to-3D-portrait generation that overcomes the aforementioned issues. To accomplish this, we train a 3D portrait generator, 3DPortraitGAN-Pyramid, as a robust prior. This generator is capable of producing 360{\deg} canonical 3D portraits, serving as a starting point for the subsequent diffusion-based generation process. To mitigate the "grid-like" artifact caused by the high-frequency information in the feature-map-based 3D representation commonly used by most 3D-aware GANs, we integrate a novel pyramid tri-grid 3D representation into 3DPortraitGAN-Pyramid. To generate 3D portraits from text, we first project a randomly generated image aligned with the given prompt into the pre-trained 3DPortraitGAN-Pyramid's latent space. The resulting latent code is then used to synthesize a pyramid tri-grid. Beginning with the obtained pyramid tri-grid, we use score distillation sampling to distill the diffusion model's knowledge into the pyramid tri-grid. Following that, we utilize the diffusion model to refine the rendered images of the 3D portrait and then use these refined images as training data to further optimize the pyramid tri-grid, effectively eliminating issues with unrealistic color and unnatural artifacts. Our experimental results show that Portrait3D can produce realistic, high-quality, and canonical 3D portraits that align with the prompt.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Single-Image 3D Human Digitization with Shape-Guided Diffusion. In SIGGRAPH Asia 2023 Conference Papers. Article 62, 11 pages.
  2. imGHUM: Implicit Generative Models of 3D Human Shape and Articulated Pose. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 5461–5470.
  3. PanoHead: Geometry-Aware 3D Full-Head Synthesis in 360deg. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR. 20950–20959.
  4. Efficient Geometry-aware 3D Generative Adversarial Networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR. 16102–16112.
  5. Pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR. 5799–5809.
  6. Mimic3D: Thriving 3D-Aware GANs via 3D-to-2D Imitation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2338–2348.
  7. MonoGaussianAvatar: Monocular Gaussian Point-based Head Avatar. CoRR abs/2312.04558 (2023).
  8. GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images. In Advances in Neural Information Processing Systems, Vol. 35. 31841–31854.
  9. Generative Adversarial Nets. In Advances in Neural Information Processing Systems, Vol. 27. 2672–2680.
  10. StyleNeRF: A Style-based 3D Aware Generator for High-resolution Image Synthesis. In The 10th International Conference on Learning Representations, ICLR.
  11. DensePose: Dense Human Pose Estimation in the Wild. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR. 7297–7306.
  12. Improved Training of Wasserstein GANs. In Advances in Neural Information Processing Systems, Vol. 30. 5767–5777.
  13. HeadSculpt: Crafting 3D Head Avatars with Text. CoRR abs/2306.03038 (2023).
  14. Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023. IEEE, 19683–19693.
  15. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP. 7514–7528.
  16. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Advances in Neural Information Processing Systems, Vol. 30. 6626–6637.
  17. SiTH: Single-view Textured Human Reconstruction with Image-Conditioned Diffusion. arXiv:2311.15855 [cs.CV]
  18. Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems, Vol. 33. 6840–6851.
  19. HumanLiff: Layer-wise 3D Human Generation with Diffusion Model. CoRR abs/2308.09712 (2023).
  20. HumanNorm: Learning Normal Diffusion Model for High-quality and Realistic 3D Human Generation. CoRR abs/2310.01406 (2023).
  21. arXiv:2308.08545 [cs.CV]
  22. AvatarCraft: Transforming Text into Neural Human Avatars with Parameterized Shape and Pose Control. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 14371–14382.
  23. Progressive Growing of GANs for Improved Quality, Stability, and Variation. In 6th International Conference on Learning Representations, ICLR.
  24. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Trans. Graph. 42, 4 (2023), 139:1–139:14.
  25. 3D GAN Inversion with Pose Optimization. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2023, Waikoloa, HI, USA, January 2-7, 2023. 2966–2975.
  26. DreamHuman: Animatable 3D Avatars from Text. CoRR abs/2306.09329 (2023).
  27. Learning a model of facial shape and expression from 4D scans. ACM Trans. Graph. 36, 6 (2017), 194:1–194:17.
  28. LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching. CoRR abs/2311.11284 (2023).
  29. TADA! Text to Animatable Digital Avatars. CoRR abs/2308.10899 (2023).
  30. HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusion. CoRR abs/2310.08579 (2023).
  31. HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splatting. CoRR abs/2311.17061 (2023).
  32. SMPL: a skinned multi-person linear model. ACM Trans. Graph. 34, 6 (2015), 248:1–248:16.
  33. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I (Lecture Notes in Computer Science, Vol. 12346). 405–421.
  34. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. 41, 4 (2022), 102:1–102:15.
  35. BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images. In Advances in Neural Information Processing Systems, Vol. 33. 6767–6778.
  36. DreamFusion: Text-to-3D using 2D Diffusion. In The 11th International Conference on Learning Representations, ICLR.
  37. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. In 4th International Conference on Learning Representations, ICLR.
  38. Rig3DGS: Creating Controllable Portraits from Casual Monocular Videos. CoRR abs/2402.03723 (2024).
  39. High-Resolution Image Synthesis With Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10684–10695.
  40. VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids. In Advances in Neural Information Processing Systems, Vol. 35. 33999–34011.
  41. EpiGRAF: Rethinking training of 3D GANs. CoRR abs/2206.10535 (2022). https://doi.org/10.48550/arXiv.2206.10535 arXiv:2206.10535
  42. Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars. CoRR abs/2211.11208 (2022).
  43. Real-Time Radiance Fields for Single-Image Portrait View Synthesis. ACM Trans. Graph. 42, 4 (2023), 135:1–135:15.
  44. What You See is What You GAN: Rendering Every Pixel for High-Fidelity Geometry in 3D GANs. CoRR abs/2401.02411 (2024).
  45. GaussianHead: High-fidelity Head Avatars with Learnable Gaussian Derivation. arXiv:2312.01632 [cs.CV]
  46. NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. In Advances in Neural Information Processing Systems, Vol. 34. 27171–27183.
  47. RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 4563–4573.
  48. ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation. In Advances in Neural Information Processing Systems, Vol. 34.
  49. 3DPortraitGAN: Learning One-Quarter Headshot 3D GANs from a Single-View Portrait Dataset with Diverse Body Poses. arXiv:2307.14770 [cs.CV]
  50. AniPortraitGAN: Animatable 3D Portrait Generation from 2D Image Collections. In SIGGRAPH Asia 2023 Conference Papers, SA 2023, Sydney, NSW, Australia, December 12-15, 2023, June Kim, Ming C. Lin, and Bernd Bickel (Eds.). ACM, 51:1–51:9.
  51. High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023. IEEE, 321–331.
  52. Gaussian Head Avatar: Ultra High-fidelity Head Avatar via Dynamic Gaussians. CoRR abs/2312.03029 (2023).
  53. SEEAvatar: Photorealistic Text-to-3D Avatar Generation with Constrained Geometry and Appearance. arXiv:2312.08889 [cs.CV]
  54. Progressive Text-to-3D Generation for Automatic 3D Prototyping. CoRR abs/2309.14600 (2023).
  55. 3D GAN Inversion with Facial Symmetry Prior. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023. IEEE, 342–351.
  56. AvatarVerse: High-quality & Stable 3D Avatar Creation from Text and Pose. CoRR abs/2308.03610 (2023).
  57. TECA: Text-Guided Generation and Editing of Compositional 3D Avatars.
  58. AvatarStudio: High-fidelity and Animatable 3D Avatar Creation from Text. CoRR abs/2311.17917 (2023).
  59. GETAvatar: Generative Textured Meshes for Animatable Human Avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2273–2282.
  60. HeadStudio: Text to Animatable Head Avatars with 3D Gaussian Splatting. CoRR abs/2402.06149 (2024).
  61. Visual Object Networks: Image Generation with Disentangled 3D Representations. In Advances in Neural Information Processing Systems, Vol. 31. 118–129.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Yiqian Wu (13 papers)
  2. Hao Xu (351 papers)
  3. Xiangjun Tang (10 papers)
  4. Xien Chen (3 papers)
  5. Siyu Tang (88 papers)
  6. Zhebin Zhang (8 papers)
  7. Chen Li (387 papers)
  8. Xiaogang Jin (38 papers)
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube