Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

HISR: Hybrid Implicit Surface Representation for Photorealistic 3D Human Reconstruction (2312.17192v1)

Published 28 Dec 2023 in cs.CV

Abstract: Neural reconstruction and rendering strategies have demonstrated state-of-the-art performances due, in part, to their ability to preserve high level shape details. Existing approaches, however, either represent objects as implicit surface functions or neural volumes and still struggle to recover shapes with heterogeneous materials, in particular human skin, hair or clothes. To this aim, we present a new hybrid implicit surface representation to model human shapes. This representation is composed of two surface layers that represent opaque and translucent regions on the clothed human body. We segment different regions automatically using visual cues and learn to reconstruct two signed distance functions (SDFs). We perform surface-based rendering on opaque regions (e.g., body, face, clothes) to preserve high-fidelity surface normals and volume rendering on translucent regions (e.g., hair). Experiments demonstrate that our approach obtains state-of-the-art results on 3D human reconstructions, and also shows competitive performances on other objects.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. Large-Scale Data for Multiple-View Stereopsis. International Journal of Computer Vision, 1–16.
  2. Photorealistic Monocular 3D Reconstruction of Humans Wearing Clothing. In IEEE Conference on Computer Vision and Pattern Recognition.
  3. Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. In IEEE International Conference on Computer Vision.
  4. Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. CVPR.
  5. Structure-Aware Sparse-View X-ray 3D Reconstruction. arXiv preprint arXiv:2311.10959.
  6. Hallucinated neural radiance fields in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12943–12952.
  7. Learning implicit fields for generative shape modeling. In IEEE Conference on Computer Vision and Pattern Recognition.
  8. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8649–8658.
  9. SurfelNeRF: Neural Surfel Radiance Fields for Online Photorealistic Reconstruction of Indoor Scenes. arXiv preprint arXiv:2304.08971.
  10. Implicit geometric regularization for learning shapes. arXiv preprint arXiv:2002.10099.
  11. ARCH: Animatable Reconstruction of Clothed Humans. In IEEE Conference on Computer Vision and Pattern Recognition.
  12. HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion. ACM Transactions on Graphics (TOG), 42(4): 1–12.
  13. AligNeRF: High-Fidelity Neural Radiance Fields via Alignment-Aware Training. arXiv preprint arXiv:2211.09682.
  14. Ray tracing volume densities. ACM Transactions on Graphics, 18(3): 165–174.
  15. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions on Graphics, 42(4).
  16. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  17. Robust 3D Self-portraits in Seconds. In IEEE Conference on Computer Vision and Pattern Recognition.
  18. Robust High-Resolution Video Matting with Temporal Guidance. arXiv preprint arXiv:2108.11515.
  19. Deep appearance models for face rendering. ACM Transactions on Graphics, 37(4): 1–13.
  20. Marching cubes: A high resolution 3D surface construction algorithm. ACM Siggraph Computer Graphics, 21(4): 163–169.
  21. Occupancy networks: Learning 3d reconstruction in function space. In IEEE Conference on Computer Vision and Pattern Recognition.
  22. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In European Conference on Computer Vision.
  23. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics, 41(4): 1–15.
  24. Neural articulated radiance field. In IEEE International Conference on Computer Vision.
  25. UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction. In IEEE International Conference on Computer Vision.
  26. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In IEEE Conference on Computer Vision and Pattern Recognition.
  27. Deepsdf: Learning continuous signed distance functions for shape representation. In IEEE Conference on Computer Vision and Pattern Recognition.
  28. Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans. In IEEE Conference on Computer Vision and Pattern Recognition.
  29. Super-resolution 3D Human Shape from a Single Low-Resolution Image. In European Conference on Computer Vision.
  30. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10318–10327.
  31. PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization. In IEEE International Conference on Computer Vision.
  32. PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization. In IEEE Conference on Computer Vision and Pattern Recognition.
  33. DeepCloth: Neural Garment Representation for Shape and Style Editing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2): 1581–1593.
  34. Nemo: Neural mesh models of contrastive features for robust 3d pose estimation. arXiv preprint arXiv:2101.12378.
  35. Neural Textured Deformable Meshes for Robust Analysis-by-Synthesis. arXiv preprint arXiv:2306.00118.
  36. VoGE: a differentiable volume renderer using gaussian ellipsoids for analysis-by-synthesis. In The Eleventh International Conference on Learning Representations.
  37. Benchmarking robustness in neural radiance fields. arXiv preprint arXiv:2301.04075.
  38. NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. In Annual Conference on Neural Information Processing Systems.
  39. Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion. arXiv preprint arXiv:2212.06135.
  40. NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction. arXiv preprint arXiv:2212.05231.
  41. Humannerf: Free-viewpoint rendering of moving people from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16210–16220.
  42. Westover, L. 1990. Footprint evaluation for volume rendering. In Proceedings of the 17th annual conference on Computer graphics and interactive techniques, 367–376.
  43. Point-nerf: Point-based neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5438–5448.
  44. Volume Rendering of Neural Implicit Surfaces. In Annual Conference on Neural Information Processing Systems.
  45. BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis. arXiv preprint arXiv:2302.14859.
  46. Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance. In Annual Conference on Neural Information Processing Systems.
  47. Yen-Chen, L. 2020. NeRF-pytorch. https://github.com/yenchenlin/nerf-pytorch/.
  48. EWA volume splatting. In IEEE Conference on Visualization.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now