Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 189 tok/s
Gemini 2.5 Pro 53 tok/s Pro
GPT-5 Medium 36 tok/s Pro
GPT-5 High 36 tok/s Pro
GPT-4o 75 tok/s Pro
Kimi K2 160 tok/s Pro
GPT OSS 120B 443 tok/s Pro
Claude Sonnet 4.5 37 tok/s Pro
2000 character limit reached

INFAMOUS-NeRF: ImproviNg FAce MOdeling Using Semantically-Aligned Hypernetworks with Neural Radiance Fields (2312.16197v1)

Published 23 Dec 2023 in cs.CV and cs.LG

Abstract: We propose INFAMOUS-NeRF, an implicit morphable face model that introduces hypernetworks to NeRF to improve the representation power in the presence of many training subjects. At the same time, INFAMOUS-NeRF resolves the classic hypernetwork tradeoff of representation power and editability by learning semantically-aligned latent spaces despite the subject-specific models, all without requiring a large pretrained model. INFAMOUS-NeRF further introduces a novel constraint to improve NeRF rendering along the face boundary. Our constraint can leverage photometric surface rendering and multi-view supervision to guide surface color prediction and improve rendering near the surface. Finally, we introduce a novel, loss-guided adaptive sampling method for more effective NeRF training by reducing the sampling redundancy. We show quantitatively and qualitatively that our method achieves higher representation power than prior face modeling methods in both controlled and in-the-wild settings. Code and models will be released upon publication.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (67)
  1. https://github.com/DCGM/ffhq-features-dataset.
  2. HyperStyle: StyleGAN inversion with HyperNetworks for real image editing. In CVPR, 2022.
  3. PanoHead: Geometry-aware 3D full-head synthesis in 360deg. In CVPR, 2023.
  4. RigNeRF: Fully controllable neural 3D portraits. In CVPR, 2022.
  5. Monocular reconstruction of neural face reflectance fields. In CVPR, 2021.
  6. Deep facial non-rigid multi-view stereo. In CVPR, 2020.
  7. Riggable 3D face reconstruction via in-network optimization. In CVPR, 2021.
  8. A morphable model for the synthesis of 3D faces. In SIGGRAPH, 1999.
  9. pi-GAN: Periodic implicit generative adversarial networks for 3D-aware image synthesis. In CVPR, 2021.
  10. Efficient geometry-aware 3D generative adversarial networks. In CVPR, 2022.
  11. ILVR: Conditioning method for denoising diffusion probabilistic models. In ICCV, 2021.
  12. GRAM: Generative radiance manifolds for 3D-aware image generation. In CVPR, 2022.
  13. DiffusionRig: Learning personalized priors for facial appearance editing. In CVPR, 2023.
  14. HyperInverter: Improving StyleGAN inversion via Hypernetwork. In CVPR, 2022.
  15. Quantitative manipulation of custom attributes on 3D-aware image synthesis. In CVPR, 2023.
  16. Learning an animatable detailed 3D face model from in-the-wild images. In SIGGRAPH, 2021.
  17. Dynamic neural radiance fields for monocular 4D facial avatar reconstruction. In CVPR, 2021.
  18. 3DMM-RF: Convolutional radiance fields for 3D face modeling. In WACV, 2023.
  19. Portrait neural radiance fields from a single image. arXiv preprint arXiv:2012.05903, 2020.
  20. Neural head avatars from monocular rgb videos. In CVPR, 2022.
  21. StyleNeRF: A style-based 3D aware generator for high-resolution image synthesis. In ICLR, 2022.
  22. HyperNetworks. In ICLR, 2017.
  23. Denoising diffusion probabilistic models. In NeurIPS, 2020.
  24. HeadNeRF: A real-time NeRF-based parametric head model. In CVPR, 2022.
  25. Face relighting with geometrically consistent shadows. In CVPR, 2022.
  26. Disentangled representation learning for 3D face shape. In CVPR, 2019.
  27. Robust egocentric photo-realistic facial expression transfer for virtual reality. In CVPR, 2022.
  28. Progressive growing of gans for improved quality, stability, and variation. In International Conference on Learning Representations, 2018.
  29. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
  30. AdaFace: Quality adaptive margin for face recognition. In CVPR, 2022.
  31. AU-aware 3D face reconstruction through personalized AU-specific blendshape learning. In ECCV, 2022.
  32. E3DGE: Self-supervised geometry-aware encoder for style-based 3D GAN inversion. In CVPR, 2023.
  33. Uncertainty-aware mesh decoder for high fidelity 3D face reconstruction. In CVPR, 2020.
  34. Learning a model of facial shape and expression from 4D scans. 2017.
  35. Towards high-fidelity 3D face reconstruction from in-the-wild images using graph convolutional networks. In CVPR, 2020.
  36. Deep meta functionals for shape representation. In ICCV, 2019.
  37. Disentangling features in 3D face shapes for joint face reconstruction and recognition. In CVPR, 2018a.
  38. Joint face alignment and 3d face reconstruction with application to face recognition. PAMI, 2018b.
  39. 3D face modeling from diverse raw scan data. In ICCV, 2019.
  40. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
  41. Learning physics-guided face relighting under directional light. In CVPR, 2020.
  42. GIRAFFE: Representing scenes as compositional generative neural feature fields. In CVPR, 2021.
  43. HyperSeg: Patch-wise Hypernetwork for real-time semantic segmentation. In CVPR, 2021.
  44. Nerfies: Deformable neural radiance fields. In ICCV, 2021a.
  45. HyperNeRF: A higher-dimensional representation for topologically varying neural radiance fields. TOG, 2021b.
  46. Semantic image synthesis with spatially-adaptive normalization. In CVPR, 2019.
  47. PVA: Pixel-aligned volumetric avatars. In CVPR, 2021.
  48. GRAF: Generative radiance fields for 3D-aware image synthesis. In NeurIPS, 2020.
  49. Self-supervised monocular 3D face reconstruction by occlusion-aware multi-view geometry consistency. ECCV, 2020.
  50. Improving 3D-aware image synthesis with a geometry-aware discriminator. In NeurIPS, 2022.
  51. Learning 3D-aware image synthesis with unknown pose distribution. In CVPR, 2023.
  52. HyperPocket: Generative point cloud completion. arXiv preprint arXiv:2102.05973, 2021.
  53. Nonlinear 3D face morphable model. In CVPR, 2018.
  54. On learning 3D face morphable model from in-the-wild images. PAMI, 2019.
  55. Towards high-fidelity nonlinear 3D face morphable model. In CVPR, 2019.
  56. Continual learning with hypernetworks. In ICLR, 2020.
  57. FaceVerse: a fine-grained and detail-controllable 3D face morphable model from a hybrid dataset. In CVPR, 2022.
  58. RODIN: A generative model for sculpting 3D digital avatars using diffusion. In CVPR, 2023.
  59. Learning compositional radiance fields of dynamic human heads. In CVPR, 2021.
  60. FaceScape: A large-scale high quality 3D face dataset and detailed riggable 3D face prediction. In CVPR, 2020.
  61. i3DMM: Deep implicit 3D morphable model of human heads. In CVPR, 2021.
  62. NeRFInvertor: High fidelity NeRF-GAN inversion for single-shot real image animation. In CVPR, 2023.
  63. pixelNeRF: Neural radiance fields from one or few images. In CVPR, 2021.
  64. Graph HyperNetworks for neural architecture search. In ICLR, 2019.
  65. FDNeRF: Few-shot dynamic neural radiance fields for face reconstruction and expression editing. In SIGGRAPH Asia, 2022.
  66. X-NeRF: Explicit neural radiance field for multi-scene 360° insufficient RGB-D views. In WACV, 2023.
  67. MoFaNeRF: Morphable facial neural radiance field. In ECCV, 2022.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up