Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

FLNeRF: 3D Facial Landmarks Estimation in Neural Radiance Fields (2211.11202v3)

Published 21 Nov 2022 in cs.CV and cs.GR

Abstract: This paper presents the first significant work on directly predicting 3D face landmarks on neural radiance fields (NeRFs). Our 3D coarse-to-fine Face Landmarks NeRF (FLNeRF) model efficiently samples from a given face NeRF with individual facial features for accurate landmarks detection. Expression augmentation is applied to facial features in a fine scale to simulate large emotions range including exaggerated facial expressions (e.g., cheek blowing, wide opening mouth, eye blinking) for training FLNeRF. Qualitative and quantitative comparison with related state-of-the-art 3D facial landmark estimation methods demonstrate the efficacy of FLNeRF, which contributes to downstream tasks such as high-quality face editing and swapping with direct control using our NeRF landmarks. Code and data will be available. Github link: https://github.com/ZHANG1023/FLNeRF.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (98)
  1. Locating facial features with an extended active shape model. In David A. Forsyth, Philip H. S. Torr, and Andrew Zisserman, editors, European Conference on Computer Vision (ECCV), 2008.
  2. 2nd 3d face alignment in the wild challenge, 2019.
  3. Augmented faces introduction, 2022.
  4. Blend shape interpolation and facs for realistic avatar. 3D Research, 6:1–10, 01 2015.
  5. Rignerf: Fully controllable neural 3d portraits. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  6. A morphable model for the synthesis of 3d faces. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH, pages 187–194, 1999.
  7. F.L. Bookstein. Principal warps: thin-plate splines and the decomposition of deformations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(6):567–585, 1989.
  8. How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In International Conference on Computer Vision, 2017.
  9. Efficient geometry-aware 3D generative adversarial networks. In CVPR, 2022.
  10. Voxresnet: Deep voxelwise residual networks for volumetric brain segmentation. arXiv preprint arXiv:1608.05895, 2016.
  11. Virtual elastic objects. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15827–15837, June 2022.
  12. Differentiable surface rendering via non-differentiable sampling. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  13. Active appearance models. In Hans Burkhardt and Bernd Neumann, editors, European Conference on Computer Vision (ECCV), pages 484–498, 1998.
  14. Active shape models - ’smart snakes’. In David C. Hogg and Roger Boyle, editors, British Machine Vision Conference(BMVC), 1992.
  15. Active shape models-their training and application. Comput. Vis. Image Underst., 61(1):38–59, 1995.
  16. Gram: Generative radiance manifolds for 3d-aware image generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  17. Joey deVilla. Augmented reality in android with google’s face api, 2017.
  18. Neural radiance flow for 4d view synthesis and video processing. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  19. Learning an animatable detailed 3D face model from in-the-wild images. volume 40, 2021.
  20. Wing loss for robust facial landmark localisation with convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  21. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8649–8658, June 2021.
  22. Portrait neural radiance fields from a single image. arXiv preprint arXiv:2012.05903, 2020.
  23. Fastnerf: High-fidelity neural rendering at 200fps. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  24. Towards fast, accurate and stable 3d dense face alignment. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, European Conference on Computer Vision (ECCV), pages 152–168, 2020.
  25. Object-centric neural scene rendering. arXiv preprint arXiv:2012.08503, 2020.
  26. Far-gan for one-shot face reenactment. CoRR, abs/2005.06402, 2020.
  27. Deep residual learning for image recognition. In CVPR, 2016.
  28. Baking neural radiance fields for real-time view synthesis. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  29. Headnerf: A real-time nerf-based parametric head model. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  30. Efficientnerf: Efficient neural radiance fields. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  31. Neuralfusion: Neural volumetric rendering under human-object interactions. arXiv preprint arXiv:2202.12825, 2022.
  32. Geonerf: Generalizing nerf with geometry priors. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  33. Actgan: Flexible and efficient one-shot face reenactment. In 2020 8th International Workshop on Biometrics and Forensics (IWBF), pages 1–6, 2020.
  34. Luvli face alignment: Estimating landmarks’ location, uncertainty, and visibility likelihood. CoRR, abs/2004.02980, 2020.
  35. Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  36. Structured landmark detection via topology-adapting deep graph learning. In European Conference on Computer Vision (ECCV), pages 266–283, 2020.
  37. Neural scene flow fields for space-time view synthesis of dynamic scenes. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  38. Autoint: Automatic integration for fast neural volume rendering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  39. Neural sparse voxel fields. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
  40. An intriguing failing of convolutional neural networks and the coordconv solution. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
  41. Editing conditional radiance fields. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  42. A deep regression architecture with two-stage re-initialization for high performance facial landmark detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3691–3700, 2017.
  43. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), 2020.
  44. A distinctive landmark-based face recognition system for identical twins by extracting novel weighted features. Computers & Electrical Engineering, 94:107326, 2021.
  45. Autorf: Learning 3d object radiance fields from single view observations. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  46. Stacked hourglass networks for human pose estimation. CoRR, abs/1603.06937, 2016.
  47. Giraffe: Representing scenes as compositional generative neural feature fields. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  48. Neural articulated radiance field. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  49. StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13503–13513, June 2022.
  50. Neural scene graphs for dynamic scenes. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  51. Nerfies: Deformable neural radiance fields. IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  52. Animatable neural radiance fields for modeling dynamic human bodies. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  53. The 2nd 3d face alignment in the wild challenge (3dfaw-video): Dense reconstruction from video. 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 3082–3089, 2019.
  54. D-nerf: Neural radiance fields for dynamic scenes. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  55. Derf: Decomposed radiance fields. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  56. Lolnerf: Learn from one look, 2022.
  57. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  58. Plenoxels: Radiance fields without neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  59. Accurate regression procedures for active appearance models. In Jesse Hoey, Stephen J. McKenna, and Emanuele Trucco, editors, British Machine Vision Conference, BMVC 2011, Dundee, UK, August 29 - September 2, 2011. Proceedings, pages 1–11. BMVA Press, 2011.
  60. Doublefield: Bridging the neural surface and radiance fields for high-fidelity human reconstruction and rendering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  61. A novel face recognition using specific values from deep neural network-based landmarks. In 2023 IEEE International Conference on Consumer Electronics (ICCE), pages 1–6, 2023.
  62. 3d landmark-based face restoration for recognition using variational autoencoder and triplet loss. IET Biom., 10(1):87–98, 2021.
  63. Improving 3d-aware image synthesis with a geometry-aware discriminator. In NeurIPS, 2022.
  64. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  65. Efficient and accurate face alignment by global regression and cascaded local refinement. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2019, Long Beach, CA, USA, June 16-20, 2019, pages 267–276. Computer Vision Foundation / IEEE, 2019.
  66. Light field neural rendering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  67. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  68. Fenerf: Face editing in neural radiance fields. arXiv preprint arXiv:2111.15490, 2021.
  69. Deep high-resolution representation learning for human pose estimation. CoRR, abs/1902.09212, 2019.
  70. Deep convolutional network cascade for facial point detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3476–3483. IEEE Computer Society, 2013.
  71. Learned initializations for optimizing coordinate-based neural representations. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  72. Mnemonic descent method: A recurrent process applied for end-to-end face alignment. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4177–4187, 2016.
  73. Mnemonic descent method: A recurrent process applied for end-to-end face alignment. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4177–4187. IEEE Computer Society, 2016.
  74. A deeply-initialized coarse-to-fine ensemble of regression trees for face alignment. In European Conference on Computer Vision (ECCV), 2018.
  75. Face transfer with multilinear models. In John W. Finnegan and Dave Shreiner, editors, International Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2006, Boston, Massachusetts, USA, July 30 - August 3, 2006, Courses, page 24. ACM, 2006.
  76. Fourier plenoctrees for dynamic radiance field rendering in real-time. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13524–13534, June 2022.
  77. HumanNeRF: Free-viewpoint rendering of moving people from monocular video. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16210–16220, June 2022.
  78. Synergy between 3dmm and 3d landmarks for accurate 3d facial geometry. In 3DV, 2021.
  79. Look at boundary: A boundary-aware face alignment algorithm. CoRR, abs/1805.10483, 2018.
  80. Sparse local patch transformer for robust face alignment and landmarks inherent relation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4052–4061, June 2022.
  81. Space-time neural irradiance fields for free-viewpoint video. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  82. Point-nerf: Point-based neural radiance fields. arXiv preprint arXiv:2201.08845, 2022.
  83. Surface-aligned neural radiance fields for controllable 3d human synthesis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  84. Learning object-compositional neural radiance field for editable scene rendering. In IEEE/CVF International Conference on Computer Vision (ICCV), October 2021.
  85. Banmo: Building animatable 3d neural models from many casual videos. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  86. Mmface: A multi-metric regression network for unconstrained face reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 7663–7672. Computer Vision Foundation / IEEE, 2019.
  87. PlenOctrees for real-time rendering of neural radiance fields. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  88. Star: Self-supervised tracking and reconstruction of rigid objects in motion with neural rendering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13144–13152, 2021.
  89. HiuKim Yuen. Augmented reality in android with google’s face api, 2022.
  90. Few-shot adversarial learning of realistic neural talking head models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  91. Nerf++: Analyzing and improving neural radiance fields. arXiv:2010.07492, 2020.
  92. Humannerf: Efficiently generated human radiance field from sparse inputs. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7743–7753, June 2022.
  93. Structured local radiance fields for human avatar modeling. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
  94. Facescape: 3d facial dataset and benchmark for single-view 3d face reconstruction. arXiv preprint arXiv:2111.01082, 2021.
  95. Robust facial landmark detection via occlusion-adaptive deep networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3481–3491, 2019.
  96. Face alignment by coarse-to-fine shape searching. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4998–5006. IEEE Computer Society, 2015.
  97. Face alignment in full pose range: A 3d total solution. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(1):78–92, jan 2019.
  98. Mofanerf: Morphable facial neural radiance field. In European Conference on Computer Vision (ECCV), 2022.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - ZHANG1023/FLNeRF (17 stars)