Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 172 tok/s
Gemini 2.5 Pro 49 tok/s Pro
GPT-5 Medium 38 tok/s Pro
GPT-5 High 30 tok/s Pro
GPT-4o 73 tok/s Pro
Kimi K2 231 tok/s Pro
GPT OSS 120B 427 tok/s Pro
Claude Sonnet 4.5 38 tok/s Pro
2000 character limit reached

Hybrid Explicit Representation for Ultra-Realistic Head Avatars (2403.11453v2)

Published 18 Mar 2024 in cs.GR and cs.CV

Abstract: We introduce a novel approach to creating ultra-realistic head avatars and rendering them in real-time (>30fps at $2048 \times 1334$ resolution). First, we propose a hybrid explicit representation that combines the advantages of two primitive-based efficient rendering techniques. UV-mapped 3D mesh is utilized to capture sharp and rich textures on smooth surfaces, while 3D Gaussian Splatting is employed to represent complex geometric structures. In the pipeline of modeling an avatar, after tracking parametric models based on captured multi-view RGB videos, our goal is to simultaneously optimize the texture and opacity map of mesh, as well as a set of 3D Gaussian splats localized and rigged onto the mesh facets. Specifically, we perform $\alpha$-blending on the color and opacity values based on the merged and re-ordered z-buffer from the rasterization results of mesh and 3DGS. This process involves the mesh and 3DGS adaptively fitting the captured visual information to outline a high-fidelity digital avatar. To avoid artifacts caused by Gaussian splats crossing the mesh facets, we design a stable hybrid depth sorting strategy. Experiments illustrate that our modeled results exceed those of state-of-the-art approaches.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Ang Cao and Justin Johnson. 2023. Hexplane: A fast representation for dynamic scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 130–141.
  2. Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction. In Proceedings of the European Conference on Computer Vision. Springer, 608––625.
  3. Efficient geometry-aware 3D generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 16123–16133.
  4. Tensorf: Tensorial radiance fields. In Proceedings of the European Conference on Computer Vision. Springer, 333–350.
  5. UV Volumes for real-time rendering of editable free-view human performance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 16621–16631.
  6. Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 16569–16578.
  7. High-quality streamable free-viewpoint video. ACM Transactions on Graphics 34, 4 (2015).
  8. Neural Radiance Flow for 4D View Synthesis and Video Processing. In Proceedings of the IEEE International Conference on Computer Vision. 14304–14314.
  9. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Papers. 1–9.
  10. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 12479–12488.
  11. Plenoxels: Radiance Fields without Neural Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5491–5500.
  12. Local Deep Implicit Functions for 3D Shape. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4856–4865.
  13. Baking neural radiance fields for real-time view synthesis. In Proceedings of the IEEE International Conference on Computer Vision. 5875–5884.
  14. Local Implicit Grid Representations for 3D Scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, Los Alamitos, CA, USA, 6000–6009.
  15. Neural 3D Mesh Renderer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3907–3916.
  16. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions on Graphics 42, 4 (2023).
  17. Modular primitives for high-performance differentiable rendering. ACM Transactions on Graphics 39, 6 (2020), 1–14.
  18. Neural sparse voxel fields. Proceedings of the Advances in Neural Information Processing Systems 33 (2020), 15651–15663.
  19. Neural volumes: learning dynamic renderable volumes from images. ACM Transactions on Graphics 38, 4 (2019).
  20. Mixture of volumetric primitives for efficient neural rendering. ACM Transactions on Graphics 40, 4 (2021), 1–13.
  21. Matthew M Loper and Michael J Black. 2014. OpenDR: An approximate differentiable renderer. In Proceedings of the European Conference on Computer Vision. 154–169.
  22. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint arXiv:2308.09713 (2023).
  23. Occupancy Networks: Learning 3D Reconstruction in Function Space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4455–4465.
  24. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In Proceedings of the IEEE International Conference on Computer Vision.
  25. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics 41, 4 (2022), 1–15.
  26. Extracting Triangular 3D Models, Materials, and Lighting From Images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8280–8290.
  27. Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics. In Proceedings of the IEEE International Conference on Computer Vision. 5378–5388.
  28. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 165–174.
  29. Nerfies: Deformable Neural Radiance Fields. In Proceedings of the IEEE International Conference on Computer Vision. 5845–5854.
  30. HyperNeRF: a higher-dimensional representation for topologically varying neural radiance fields. ACM Transactions on Graphics 40, 6 (2021).
  31. Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies. In Proceedings of the IEEE International Conference on Computer Vision.
  32. Convolutional Occupancy Networks. In Proceedings of the European Conference on Computer Vision. 523–540.
  33. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9054–9063.
  34. D-NeRF: Neural Radiance Fields for Dynamic Scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10313–10322.
  35. Accelerating 3D Deep Learning with PyTorch3D. arXiv:2007.08501 (2020).
  36. PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization. In Proceedings of the IEEE International Conference on Computer Vision. 2304–2314.
  37. PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 81–90.
  38. DeepVoxels: Learning Persistent 3D Feature Embeddings. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2432–2441.
  39. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. IEEE Transactions on Visualization and Computer Graphics 29, 5 (2023), 2732–2742.
  40. A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose. In Proceedings of the Advances in Neural Information Processing Systems.
  41. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5459–5469.
  42. TensorFlow Graphics: Computer Graphics Meets Deep Learning.
  43. Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 76–87.
  44. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528 (2023).
  45. Multiface: A dataset for neural face rendering. arXiv preprint arXiv:2207.11243 (2022).
  46. Fast and robust non-rigid registration using accelerated majorization-minimization. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).
  47. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE International Conference on Computer Vision. 5752–5761.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

This paper has been mentioned in 2 tweets and received 0 likes.

Upgrade to Pro to view all of the tweets about this paper:

Start a free 7-day Pro trial