Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
140 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

SiTH: Single-view Textured Human Reconstruction with Image-Conditioned Diffusion (2311.15855v2)

Published 27 Nov 2023 in cs.CV

Abstract: A long-standing goal of 3D human reconstruction is to create lifelike and fully detailed 3D humans from single-view images. The main challenge lies in inferring unknown body shapes, appearances, and clothing details in areas not visible in the images. To address this, we propose SiTH, a novel pipeline that uniquely integrates an image-conditioned diffusion model into a 3D mesh reconstruction workflow. At the core of our method lies the decomposition of the challenging single-view reconstruction problem into generative hallucination and reconstruction subproblems. For the former, we employ a powerful generative diffusion model to hallucinate unseen back-view appearance based on the input images. For the latter, we leverage skinned body meshes as guidance to recover full-body texture meshes from the input and back-view images. SiTH requires as few as 500 3D human scans for training while maintaining its generality and robustness to diverse images. Extensive evaluations on two 3D human benchmarks, including our newly created one, highlighted our method's superior accuracy and perceptual quality in 3D textured human reconstruction. Our code and evaluation benchmark are available at https://ait.ethz.ch/sith

Definition Search Book Streamline Icon: https://streamlinehq.com
References (79)
  1. Renderpeople, https://renderpeople.com/.
  2. Stable diffusion image variations. huggingface.co/lambdalabs/stable-diffusion-image-conditioned.
  3. Single-image 3d human digitization with shape-guided diffusion. In SIGGRAPH Asia, 2023.
  4. Photorealistic monocular 3d reconstruction of humans wearing clothing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  5. Method for registration of 3-d shapes. In Sensor fusion IV: control paradigms and data structures, pages 586–606. Spie, 1992.
  6. Demystifying MMD gans. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
  7. Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image. In Proceedings of the European Conference on Computer Vision (ECCV). Springer International Publishing, 2016.
  8. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 33:1877–1901, 2020.
  9. Smpler-x: Scaling up expressive human pose and shape estimation. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
  10. Dreamavatar: Text-and-shape guided 3d human avatar generation via diffusion models. arXiv preprint arXiv:2304.00916, 2023.
  11. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2023.
  12. Structured 3d features for reconstructing relightable and animatable avatars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  13. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  14. Pina: Learning a personalized implicit neural avatar from a single rgb-d video sequence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  15. Fof: Learning fourier occupancy field for monocular real-time human reconstruction. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
  16. Densepose: Dense human pose estimation in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  17. Vid2avatar: 3d avatar reconstruction from videos in the wild via self-supervised scene decomposition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  18. High-fidelity 3d human digitization from single 2k resolution images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  19. Instruct-nerf2nerf: Editing 3d scenes with instructions. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2023.
  20. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 16000–16009, 2022.
  21. Arch++: Animation-ready clothed human reconstruction revisited. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
  22. Learning locally editable virtual humans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  23. LoRA: Low-rank adaptation of large language models. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
  24. TeCH: Text-guided Reconstruction of Lifelike Clothed Humans. In International Conference on 3D Vision (3DV), 2024.
  25. Arch: Animatable reconstruction of clothed humans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  26. Selfrecon: Self reconstruction your digital avatar from monocular video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022a.
  27. Avatarcraft: Transforming text into neural human avatars with parameterized shape and pose control. arXiv preprint arXiv:2303.17606, 2023a.
  28. Instantavatar: Learning avatars from monocular video in 60 seconds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023b.
  29. Neuman: Neural human radiance field from a single video. In Proceedings of the European Conference on Computer Vision (ECCV), 2022b.
  30. Perceptual losses for real-time style transfer and super-resolution. In Proceedings of the European Conference on Computer Vision (ECCV), pages 694–711. Springer, 2016.
  31. Dreampose: Fashion image-to-video synthesis via stable diffusion. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2023.
  32. aitviewer, 2022.
  33. Screened poisson surface reconstruction. ACM Transactions on Graphics (TOG), 32(3):1–13, 2013.
  34. Poisson surface reconstruction. In Proceedings of the fourth Eurographics symposium on Geometry processing, 2006.
  35. Chupa: Carving 3d clothed humans from skinned shape priors using 2d diffusion probabilistic models. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2023.
  36. Segment anything. arXiv:2304.02643, 2023.
  37. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  38. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning (ICML), 2022.
  39. High-Fidelity Clothed Avatar Reconstruction from a Single Image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  40. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  41. Zero-1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2023.
  42. SMPL: A skinned multi-person linear model. ACM Transactions on Graphics (TOG), 34(6):248:1–248:16, 2015.
  43. Marching cubes: A high resolution 3d surface construction algorithm, 1998.
  44. Mediapipe: A framework for perceiving and processing reality. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshop (CVPRW), 2019.
  45. Learning to Dress 3D People in Generative Clothing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  46. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  47. Latent-nerf for shape-guided generation of 3d shapes and textures. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  48. Nerf: Representing scenes as neural radiance fields for view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  49. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (TOG), 41(4):102:1–102:15, 2022.
  50. Stacked hourglass networks for human pose estimation. In Proceedings of the European Conference on Computer Vision (ECCV). Springer, 2016.
  51. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  52. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
  53. Dreamfusion: Text-to-3d using 2d diffusion. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
  54. Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint arXiv:2306.17843, 2023.
  55. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
  56. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
  57. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  58. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  59. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems (NeurIPS), 35, 2022.
  60. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019.
  61. Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  62. X-avatar: Expressive human avatars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  63. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
  64. Dinar: Diffusion inpainting of neural textures for one-shot human avatars. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2023.
  65. What do single-view 3d reconstruction networks learn? In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  66. Advances in neural rendering. In Annual Conference of the European Association for Computer Graphics (EUROGRAPHICS), pages 703–735. Wiley Online Library, 2022.
  67. Recovering 3d human mesh from monocular images: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2023.
  68. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  69. High-resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  70. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003.
  71. HumanNeRF: Free-viewpoint rendering of moving people from monocular video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 16210–16220, 2022.
  72. Neural fields in visual computing and beyond. In Annual Conference of the European Association for Computer Graphics (EUROGRAPHICS), pages 641–676. Wiley Online Library, 2022.
  73. ICON: Implicit Clothed humans Obtained from Normals. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  74. ECON: Explicit Clothed humans Optimized via Normal integration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  75. Function4d: Real-time human volumetric capture from very sparse consumer rgbd sensors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  76. Avatarverse: High-quality stable 3d avatar creation from text and pose. arXiv preprint arXiv:2308.03610, 2023a.
  77. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2023b.
  78. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  79. Pamir: Parametric model-conditioned implicit representation for image-based human reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021.
Citations (15)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube