Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

FED-NeRF: Achieve High 3D Consistency and Temporal Coherence for Face Video Editing on Dynamic NeRF (2401.02616v1)

Published 5 Jan 2024 in cs.CV

Abstract: The success of the GAN-NeRF structure has enabled face editing on NeRF to maintain 3D view consistency. However, achieving simultaneously multi-view consistency and temporal coherence while editing video sequences remains a formidable challenge. This paper proposes a novel face video editing architecture built upon the dynamic face GAN-NeRF structure, which effectively utilizes video sequences to restore the latent code and 3D face geometry. By editing the latent code, multi-view consistent editing on the face can be ensured, as validated by multiview stereo reconstruction on the resulting edited images in our dynamic NeRF. As the estimation of face geometries occurs on a frame-by-frame basis, this may introduce a jittering issue. We propose a stabilizer that maintains temporal coherence by preserving smooth changes of face expressions in consecutive frames. Quantitative and qualitative analyses reveal that our method, as the pioneering 4D face video editor, achieves state-of-the-art performance in comparison to existing 2D or 3D-based approaches independently addressing identity and motion. Codes will be released.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (56)
  1. Third time’s the charm? image and video editing with stylegan3. arXiv preprint https://arxiv.org/abs/2201.13433, 2022.
  2. A morphable model for the synthesis of 3d faces. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH, pages 187–194, 1999.
  3. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021a.
  4. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proc. CVPR, 2021b.
  5. Efficient geometry-aware 3d generative adversarial networks. CoRR, abs/2112.07945, 2021c.
  6. Video face replacement. In Proceedings of the 2011 SIGGRAPH Asia conference, pages 1–10, 2011.
  7. EMOCA: Emotion driven monocular face capture and animation. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 20311–20322, 2022.
  8. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  9. Learning an animatable detailed 3D face model from in-the-wild images. ACM Transactions on Graphics, (Proc. SIGGRAPH), 40(8), 2021.
  10. Generative adversarial networks, 2014.
  11. Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis. CoRR, abs/2110.08985, 2021.
  12. Identity-aware and shape-aware propagation of face editing in videos. IEEE Transactions on Visualization and Computer Graphics, pages 1–12, 2023.
  13. A style-based generator architecture for generative adversarial networks. CoRR, abs/1812.04948, 2018.
  14. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019a.
  15. Analyzing and improving the image quality of stylegan. CoRR, abs/1912.04958, 2019b.
  16. Analyzing and improving the image quality of StyleGAN. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  17. Alias-free generative adversarial networks. CoRR, abs/2106.12423, 2021.
  18. Diffusion video autoencoders: Toward temporally consistent face video editing via disentangled video encoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6091–6100, 2023.
  19. Adam: A method for stochastic optimization, 2017.
  20. Learning a model of facial shape and expression from 4D scans. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6):194:1–194:17, 2017.
  21. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  22. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
  23. Giraffe: Representing scenes as compositional generative neural feature fields. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
  24. Nerfies: Deformable neural radiance fields. ICCV, 2021a.
  25. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. ACM Trans. Graph., 40(6), 2021b.
  26. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2085–2094, 2021.
  27. Cagenerf: Cage-based neural radiance field for generalized 3d deformation and animation. In Advances in Neural Information Processing Systems, pages 31402–31415. Curran Associates, Inc., 2022.
  28. D-nerf: Neural radiance fields for dynamic scenes. arXiv preprint arXiv:2011.13961, 2020.
  29. Learning transferable visual models from natural language supervision. In ICML, 2021.
  30. High-resolution image synthesis with latent diffusion models, 2021.
  31. Structure-from-Motion Revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  32. GRAF: generative radiance fields for 3d-aware image synthesis. CoRR, abs/2007.02442, 2020a.
  33. Graf: Generative radiance fields for 3d-aware image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2020b.
  34. Interfacegan: Interpreting the disentangled face representation learned by gans, 2020.
  35. Very deep convolutional networks for large-scale image recognition, 2015.
  36. Implicit neural representations with periodic activation functions. CoRR, abs/2006.09661, 2020.
  37. Diffusion guided domain adaptation of image generators. arXiv preprint https://arxiv.org/abs/2212.04473, 2022.
  38. Ide-3d: Interactive disentangled editing for high-resolution 3d-aware portrait synthesis. ACM Transactions on Graphics (TOG), 41(6):1–10, 2022a.
  39. Fenerf: Face editing in neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7672–7682, 2022b.
  40. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 402–419. Springer, 2020.
  41. Face2face: Real-time face capture and reenactment of rgb videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  42. Regressing robust and discriminative 3d morphable models with a very deep neural network, 2016.
  43. Stitch it in time: Gan-based facial editing of real videos, 2022.
  44. Neural trajectory fields for dynamic novel view synthesis, 2021.
  45. Anifacegan: Animatable 3d-aware face image generation for video avatars. In Advances in Neural Information Processing Systems, 2022.
  46. Stylespace analysis: Disentangled controls for stylegan image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12863–12872, 2021.
  47. Space-time neural irradiance fields for free-viewpoint video, 2021.
  48. Omniavatar: Geometry-guided controllable 3d head synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12814–12824, 2023a.
  49. Temporally consistent semantic video editing. arXiv preprint arXiv: 2206.10590, 2022.
  50. Rigid: Recurrent gan inversion and editing of real face videos, 2023b.
  51. Face morphing using 3d-aware appearance optimization. In Graphics Interface, pages 93–99. Citeseer, 2012.
  52. A latent transformer for disentangled face editing in images and videos. 2021 International Conference on Computer Vision, 2021.
  53. Make encoder great again in 3d gan inversion through geometry and occlusion-aware encoding. arXiv preprint arXiv:2303.12326, 2023.
  54. Facednerf: Semantics-driven face reconstruction, prompt editing and relighting with diffusion models, 2023.
  55. CelebV-HQ: A large-scale video facial attributes dataset. In ECCV, 2022.
  56. Mofanerf: Morphable facial neural radiance field. In European Conference on Computer Vision, 2022.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now