Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

NECA: Neural Customizable Human Avatar (2403.10335v1)

Published 15 Mar 2024 in cs.CV

Abstract: Human avatar has become a novel type of 3D asset with various applications. Ideally, a human avatar should be fully customizable to accommodate different settings and environments. In this work, we introduce NECA, an approach capable of learning versatile human representation from monocular or sparse-view videos, enabling granular customization across aspects such as pose, shadow, shape, lighting and texture. The core of our approach is to represent humans in complementary dual spaces and predict disentangled neural fields of geometry, albedo, shadow, as well as an external lighting, from which we are able to derive realistic rendering with high-frequency details via volumetric rendering. Extensive experiments demonstrate the advantage of our method over the state-of-the-art methods in photorealistic rendering, as well as various editing tasks such as novel pose synthesis and relighting. The code is available at https://github.com/iSEE-Laboratory/NECA.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (79)
  1. Mixamo. https://www.mixamo.com/.
  2. Renderpeople. https://renderpeople.com/.
  3. Photorealistic monocular 3d reconstruction of humans wearing clothing. In CVPR, 2022.
  4. Deep 3d capture: Geometry and reflectance from sparse multi-view images. In CVPR, 2020.
  5. James F. Blinn. Simulation of wrinkled surfaces. In SIGGRAPH, 1978.
  6. Analysis of individual differences in multidimensional scaling via an n-way generalization of “eckart-young” decomposition. Psychometrika, 35:283–319, 1970.
  7. Tensorf: Tensorial radiance fields. In ECCV, 2022.
  8. Snarf: Differentiable forward skinning for animating non-rigid neural implicit shapes. In ICCV, 2021.
  9. Fast-SNARF: A fast deformer for articulated neural fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–15, 2023a.
  10. Uv volumes for real-time rendering of editable free-view human performance. In CVPR, 2023b.
  11. Relighting4d: Neural relightable human from videos. In ECCV, 2022.
  12. Learning implicit fields for generative shape modeling. In CVPR, 2019.
  13. Structured 3d features for reconstructing controllable avatars. In CVPR, 2023.
  14. Implicit geometric regularization for learning shapes. In ICML, 2020.
  15. Efficient light probes for real-time global illumination. ACM Trans. Graphics, 41(6), 2022.
  16. The relightables: Volumetric performance capture of humans with realistic relighting. ACM Trans. Graphics, 38(6), 2019.
  17. Deepcap: Monocular human performance capture using weak supervision. In CVPR, 2020.
  18. Real-time deep dynamic characters. ACM Trans. Graphics, 40(4), 2021.
  19. Richard A. Harshman. Foundations of the parafac procedure: Models and conditions for an "explanatory" multi-model factor analysis. 1970.
  20. Geo-pifu: Geometry and pixel aligned implicit functions for single-view human reconstruction. In NeurIPS, 2020.
  21. Arch++: Animation-ready clothed human reconstruction revisited. In ICCV, 2021.
  22. Learning locally editable virtual humans. In CVPR, 2023.
  23. Tri-miprf: Tri-mip representation for efficient anti-aliasing neural radiance fields. In ICCV, 2023.
  24. Arch: Animatable reconstruction of clothed humans. In CVPR, 2020.
  25. Neuman: Neural human radiance field from a single video. In ECCV, 2022.
  26. Tensoir: Tensorial inverse rendering. In CVPR, 2023.
  27. James T. Kajiya. The rendering equation. In SIGGRAPH, 1986.
  28. Ray tracing volume densities. In SIGGRAPH, 1984.
  29. Relighting humans: Occlusion-aware inverse rendering for full-body human images. ACM Trans. Graphics, 37(6), 2018.
  30. Adam: A method for stochastic optimization. In ICLR, 2015.
  31. Hybrik: A hybrid analytical-neural inverse kinematics solution for 3d human pose and shape estimation. In CVPR, 2021a.
  32. Learn to dance with aist++: Music conditioned 3d dance generation. In ICCV, 2021b.
  33. Tava: Template-free animatable volumetric actors. In ECCV, 2022.
  34. Posevocab: Learning joint-structured pose embeddings for human avatar modeling. In SIGGRAPH, 2023.
  35. Neural actor: Neural free-view synthesis of human actors with pose control. ACM Trans. Graphics, 40(6), 2021.
  36. SMPL: A skinned multi-person linear model. ACM Trans. Graphics, 34(6):248:1–248:16, 2015.
  37. Deep relightable textures: Volumetric performance capture with neural rendering. ACM Trans. Graphics, 39(6), 2020.
  38. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, 2019.
  39. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
  40. Stylesdf: High-resolution 3d-consistent image and geometry generation. In CVPR, 2022.
  41. STAR: A sparse trained articulated human body regressor. In ECCV, 2020.
  42. Total relighting: Learning to relight portraits for background replacement. ACM Trans. Graphics, 40(4), 2021.
  43. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, 2019.
  44. Expressive body capture: 3d hands, face, and body from a single image. In CVPR, 2019.
  45. Animatable neural radiance fields for modeling dynamic human bodies. In ICCV, 2021a.
  46. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR, 2021b.
  47. Animatable neural implicit surfaces for creating avatars from videos. arXiv preprint arXiv:2203.08133, 2022.
  48. Representing volumetric videos as dynamic mlp maps. In CVPR, 2023.
  49. Nerf for outdoor scene relighting. In ECCV, 2022.
  50. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In ICCV, 2019.
  51. Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization. In CVPR, 2020.
  52. On joint estimation of pose, geometry and svbrdf from a handheld scanner. In CVPR, 2020.
  53. Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In CVPR, 2023.
  54. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
  55. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In CVPR, 2021.
  56. Recent advances in implicit representation based 3d shape generation. Visual Intelligence, 2024.
  57. Neural reconstruction of relightable human model from monocular video. In ICCV, 2023.
  58. Predicting human poses via recurrent attention network. Visual Intelligence, 2023.
  59. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In NeurIPS, 2021.
  60. Arah: Animatable volume rendering of articulated human sdfs. In ECCV, 2022.
  61. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  62. Neural fields meet explicit geometric representations for inverse rendering of urban scenes. In CVPR, 2023.
  63. HumanNeRF: Free-viewpoint rendering of moving people from monocular video. In CVPR, 2022.
  64. De-nerf: Decoupled neural radiance fields for view-consistent appearance editing and high-frequency environmental relighting. In SIGGRAPH, 2023.
  65. ICON: Implicit Clothed humans Obtained from Normals. In CVPR, 2022.
  66. ECON: Explicit Clothed humans Optimized via Normal integration. In CVPR, 2023.
  67. Surface-aligned neural radiance fields for controllable 3d human synthesis. In CVPR, 2022.
  68. Relightable and animatable neural avatar from sparse-view video. In arXiv preprint arXiv:2308.07903, 2023.
  69. Multiview neural surface reconstruction by disentangling geometry and appearance. In NeurIPS, 2020.
  70. Volume rendering of neural implicit surfaces. In NeurIPS, 2021.
  71. Ye Yu and William A. P. Smith. Inverserendernet: Learning single image inverse rendering. In CVPR, 2019.
  72. MonoHuman: Animatable human neural field from monocular video. In CVPR, 2023.
  73. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In CVPR, 2021a.
  74. Ndf: Neural deformable fields for dynamic human modelling. In ECCV, 2022.
  75. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
  76. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Trans. Graphics, 40(6), 2021b.
  77. Learning visibility field for detailed 3d human reconstruction and relighting. In CVPR, 2023.
  78. Pamir: Parametric model-conditioned implicit representation for image-based human reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(6):3170–3184, 2022.
  79. Dual-space nerf: Learning animatable avatars and scene lighting in separate spaces. In 3DV, 2022.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now