Papers
Topics
Authors
Recent
Search
2000 character limit reached

OHTA: One-shot Hand Avatar via Data-driven Implicit Priors

Published 29 Feb 2024 in cs.CV | (2402.18969v1)

Abstract: In this paper, we delve into the creation of one-shot hand avatars, attaining high-fidelity and drivable hand representations swiftly from a single image. With the burgeoning domains of the digital human, the need for quick and personalized hand avatar creation has become increasingly critical. Existing techniques typically require extensive input data and may prove cumbersome or even impractical in certain scenarios. To enhance accessibility, we present a novel method OHTA (One-shot Hand avaTAr) that enables the creation of detailed hand avatars from merely one image. OHTA tackles the inherent difficulties of this data-limited problem by learning and utilizing data-driven hand priors. Specifically, we design a hand prior model initially employed for 1) learning various hand priors with available data and subsequently for 2) the inversion and fitting of the target identity with prior knowledge. OHTA demonstrates the capability to create high-fidelity hand avatars with consistent animatable quality, solely relying on a single image. Furthermore, we illustrate the versatility of OHTA through diverse applications, encompassing text-to-avatar conversion, hand editing, and identity latent space manipulation.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (86)
  1. Learning to reconstruct people in clothing from a single rgb camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1175–1186, 2019a.
  2. Tex2shape: Detailed full human body geometry from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2293–2303, 2019b.
  3. imghum: Implicit generative models of 3d human shape and articulated pose. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5461–5470, 2021.
  4. Learning personalized high quality volumetric head avatars from monocular rgb videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16890–16900, 2023.
  5. Keep it smpl: Automatic estimation of 3d human pose and shape from a single image. In The European Conference on Computer Vision, pages 561–578, 2016.
  6. Preface: A data-driven volumetric prior for few-shot ultra high-resolution face synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3402–3413, 2023.
  7. Authentic volumetric avatars from a phone scan. ACM Transactions on Graphics (TOG), 41(4):1–19, 2022.
  8. Snarf: Differentiable forward skinning for animating non-rigid neural implicit shapes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11594–11604, 2021a.
  9. Hand avatar: Free-pose hand animation and rendering from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8683–8693, 2023.
  10. Model-based 3d hand reconstruction via self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10451–10460, 2021b.
  11. Lisa: Learning implicit shape and appearance of hands. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20533–20543, 2022.
  12. Nasa neural articulated shape approximation. In The European Conference on Computer Vision, pages 612–628, 2020.
  13. Dart: Articulated hand model with diverse accessories and rich textures. Advances in Neural Information Processing Systems, 35:37055–37067, 2022a.
  14. Cyclehand: Increasing 3d pose estimation ability on in-the-wild monocular image through cyclic flow. In Proceedings of the ACM International Conference on Multimedia, pages 2452–2463, 2022b.
  15. Neural head avatars from monocular rgb videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18653–18664, 2022.
  16. Handnerf: Neural radiance fields for animatable interacting hands. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21078–21087, 2023.
  17. Geo-pifu: Geometry and pixel aligned implicit functions for single-view human reconstruction. Advances in Neural Information Processing Systems, 33:9276–9287, 2020.
  18. Arch++: Animation-ready clothed human reconstruction revisited. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11046–11056, 2021.
  19. Sherf: Generalizable human nerf from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
  20. One-shot implicit animatable avatars with model-based priors. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023a.
  21. Arch: Animatable reconstruction of clothed humans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3093–3102, 2020.
  22. Phrit: Parametric hand representation with implicit template. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14974–14984, 2023b.
  23. Selfrecon: Self reconstruction your digital avatar from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5605–5615, 2022.
  24. Instant-nvr: Instant neural volumetric rendering for human-object interactions from monocular rgbd stream. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 595–605, 2023.
  25. Whole-body human pose estimation in the wild. In The European Conference on Computer Vision, pages 196–214, 2020.
  26. Cg-nerf: Conditional generative neural radiance fields for 3d-aware image synthesis. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 724–733, 2023.
  27. Exemplar fine-tuning for 3d human model fitting towards in-the-wild 3d human pose estimation. In 2021 International Conference on 3D Vision (3DV), pages 42–52. IEEE, 2021.
  28. End-to-end recovery of human shape and pose. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7122–7131, 2018.
  29. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34:852–863, 2021.
  30. A skeleton-driven neural occupancy representation for articulated hands. In International Conference on 3D Vision, pages 11–21, 2021.
  31. Harp: Personalized hand reconstruction from a monocular rgb video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12802–12813, 2023.
  32. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  33. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023.
  34. Pare: Part attention regressor for 3d human body estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11127–11137, 2021.
  35. Learning to reconstruct 3d human pose and shape via model-fitting in the loop. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2252–2261, 2019.
  36. 360-degree textures of people in clothing from a single image. In International Conference on 3D Vision, pages 643–653, 2019.
  37. Learning a model of facial shape and expression from 4d scans. ACM Transactions on Graphics (TOG), 36(6):194–1, 2017.
  38. Piano: A parametric hand bone model from magnetic resonance imaging. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 816–822, 2021.
  39. Nimble: a non-rigid hand model with bones and muscles. ACM Transactions on Graphics (TOG), 41(4):1–16, 2022.
  40. Microsoft coco: Common objects in context. In The European Conference on Computer Vision, pages 740–755, 2014.
  41. Consistent123: One image to highly consistent 3d asset using case-aware diffusion priors. arXiv preprint arXiv:2309.17261, 2023.
  42. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 2023a.
  43. Zero-1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9298–9309, 2023b.
  44. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7708–7717, 2019.
  45. Mixture of volumetric primitives for efficient neural rendering. ACM Transactions on Graphics (TOG), 40(4):1–13, 2021.
  46. Smpl: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 851–866. 2023.
  47. Learning to dress 3d people in generative clothing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6469–6478, 2020.
  48. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4460–4470, 2019.
  49. Leap: Learning articulated occupancy of people. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10461–10471, 2021.
  50. Coap: Compositional articulated occupancy of people. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13201–13210, 2022.
  51. Nerf: Representing scenes as neural radiance fields for view synthesis. In The European Conference on Computer Vision, pages 405–421, 2020.
  52. Gyeongsik Moon. Bringing inputs to shared domains for 3d interacting hands recovery in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17028–17037, 2023.
  53. Interhand2. 6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image. In The European Conference on Computer Vision, pages 548–564, 2020.
  54. Actorsnerf: Animatable few-shot human rendering with generalizable nerfs. pages 18391–18401, 2023.
  55. Livehand: Real-time and photorealistic neural hand rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
  56. Star: Sparse trained articulated human body regressor. In The European Conference on Computer Vision, pages 598–613, 2020.
  57. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10975–10985, 2019.
  58. Animatable neural radiance fields for modeling dynamic human bodies. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14314–14323, 2021a.
  59. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9054–9063, 2021b.
  60. Handy: Towards a high fidelity 3d hand shape and appearance model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4670–4680, 2023.
  61. Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. In The 12th International Conference on Learning Representations, 2024.
  62. Html: A parametric hand texture model for 3d hand reconstruction and personalization. In The European Conference on Computer Vision, pages 54–71. Springer, 2020.
  63. Novel-view synthesis and pose estimation for hand-object interaction from sparse views. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15100–15111, 2023.
  64. Decoupled iterative refinement framework for interacting hands reconstruction from a single rgb image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8014–8025, 2023.
  65. Embodied hands: modeling and capturing hands and bodies together. ACM Transactions on Graphics (TOG), 36(6):245:1–245:17, 2017.
  66. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2304–2314, 2019.
  67. Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 84–93, 2020.
  68. Scanimate: Weakly supervised learning of skinned clothed avatar networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2886–2897, 2021.
  69. A-nerf: Articulated neural radiance fields for learning human shape, appearance, and pose. Advances in Neural Information Processing Systems, 34:12278–12291, 2021.
  70. Dinar: Diffusion inpainting of neural textures for one-shot human avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7062–7072, 2023.
  71. Neural-gif: Neural generalized implicit functions for animating people in clothing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11708–11718, 2021.
  72. Metaavatar: Learning animatable clothed human models from few depth images. Advances in Neural Information Processing Systems, 34:2810–2822, 2021.
  73. Mask-pose cascaded cnn for 2d hand pose estimation from single color image. IEEE Transactions on Circuits and Systems for Video Technology, 29(11):3258–3268, 2018.
  74. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  75. Humannerf: Free-viewpoint rendering of moving people from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16210–16220, 2022.
  76. De-nerf: Decoupled neural radiance fields for view-consistent appearance editing and high-frequency environmental relighting. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1–11, 2023.
  77. H-nerf: Neural radiance fields for rendering and temporal reconstruction of humans in motion. Advances in Neural Information Processing Systems, 34:14955–14966, 2021.
  78. Neumesh: Learning disentangled neural mesh-based implicit field for geometry and texture editing. In The European Conference on Computer Vision, pages 597–614, 2022.
  79. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578–4587, 2021.
  80. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023.
  81. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 586–595, 2018.
  82. Humannerf: Efficiently generated human radiance field from sparse inputs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7743–7753, 2022.
  83. Pamir: Parametric model-conditioned implicit representation for image-based human reconstruction. IEEE transactions on pattern analysis and machine intelligence, 44(6):3170–3184, 2021.
  84. Structured local radiance fields for human avatar modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15893–15903, 2022.
  85. Detailed human shape estimation from a single image by hierarchical mesh deformation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4491–4500, 2019.
  86. Contrastive representation learning for hand shape estimation. In DAGM German Conference on Pattern Recognition, pages 250–264. Springer, 2021.
Citations (3)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up