Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Monocular Identity-Conditioned Facial Reflectance Reconstruction (2404.00301v1)

Published 30 Mar 2024 in cs.CV

Abstract: Recent 3D face reconstruction methods have made remarkable advancements, yet there remain huge challenges in monocular high-quality facial reflectance reconstruction. Existing methods rely on a large amount of light-stage captured data to learn facial reflectance models. However, the lack of subject diversity poses challenges in achieving good generalization and widespread applicability. In this paper, we learn the reflectance prior in image space rather than UV space and present a framework named ID2Reflectance. Our framework can directly estimate the reflectance maps of a single image while using limited reflectance data for training. Our key insight is that reflectance data shares facial structures with RGB faces, which enables obtaining expressive facial prior from inexpensive RGB data thus reducing the dependency on reflectance data. We first learn a high-quality prior for facial reflectance. Specifically, we pretrain multi-domain facial feature codebooks and design a codebook fusion method to align the reflectance and RGB domains. Then, we propose an identity-conditioned swapping module that injects facial identity from the target image into the pre-trained autoencoder to modify the identity of the source reflectance image. Finally, we stitch multi-view swapped reflectance images to obtain renderable assets. Extensive experiments demonstrate that our method exhibits excellent generalization capability and achieves state-of-the-art facial reflectance reconstruction results for in-the-wild faces. Our project page is https://xingyuren.github.io/id2reflectance/.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (77)
  1. Ffhq-uv: Normalized facial uv-texture dataset for 3d face reconstruction. In CVPR, 2023.
  2. Riggable 3D face reconstruction via in-network optimization. In CVPR, 2021.
  3. Face swapping: automatically replacing faces in photographs. In SIGGRAPH, 2008.
  4. A morphable model for the synthesis of 3d faces. In SIGGRAPH, 1999.
  5. A 3d morphable model learnt from 10,000 faces. In CVPR, 2016.
  6. Vggface2: A dataset for recognising faces across pose and age. In FG, 2018.
  7. Efficient geometry-aware 3d generative adversarial networks. In CVPR, 2022.
  8. Simswap: An efficient framework for high fidelity face swapping. In ACM MM, 2020.
  9. Acquiring the reflectance field of a human face. In Computer Graphics and Interactive Techniques, 2000.
  10. Uv-gan: Adversarial facial uv map completion for pose-invariant face recognition. In CVPR, 2018.
  11. Arcface: Additive angular margin loss for deep face recognition. In CVPR, 2019a.
  12. Retinaface: Single-shot multi-level face localisation in the wild. In CVPR, 2020.
  13. Accurate 3D face reconstruction with weakly-supervised learning: From single image to image set. In CVPRW, 2019b.
  14. Towards high fidelity monocular face reconstruction with rich reflectance using self-supervised learning and ray tracing. In ICCV, 2021.
  15. 3D morphable face models - past, present, and future. TOG, 2020.
  16. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
  17. Towards racially unbiased skin tone estimation via scene disambiguation. In ECCV, 2022.
  18. Learning an animatable detailed 3d face model from in-the-wild images. TOG, 2021.
  19. Information bottleneck disentanglement for identity swapping. In CVPR, 2021.
  20. Ganfit: Generative adversarial network fitting for high fidelity 3d face reconstruction. In CVPR, 2019.
  21. Synthesizing coupled 3d face modalities by trunk-branch generative adversarial networks. In ECCV, 2020.
  22. Multiview face capture using polarized spherical gradient illumination. TOG, 2011.
  23. Generative adversarial nets. In NeurIPS, 2014.
  24. Sample and computation redistribution for efficient face detection. In ICLR, 2022.
  25. Learning a 3d morphable face reflectance model from low-cost data. In CVPR, 2023a.
  26. Learning a 3d morphable face reflectance model from low-cost data. In CVPR, 2023b.
  27. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV, 2017.
  28. Image-to-image translation with conditional adversarial networks. In CVPR, 2017.
  29. Progressive growing of gans for improved quality, stability, and variation. In ICLR, 2018.
  30. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
  31. Analyzing and improving the image quality of stylegan. In CVPR, 2020.
  32. Adam: A method for stochastic optimization. In ICLR, 2015.
  33. AvatarMe: Realistically renderable 3D facial reconstruction. In CVPR, 2020.
  34. Avatarme++absent{}^{++}start_FLOATSUPERSCRIPT + + end_FLOATSUPERSCRIPT: Facial shape and BRDF inference with photorealistic rendering-aware GANs. TPAMI, 2021.
  35. Fitme: Deep photorealistic 3d morphable model avatars. In CVPR, 2023.
  36. Maskgan: Towards diverse and interactive facial image manipulation. In CVPR, 2020.
  37. Uncertainty-aware mesh decoder for high fidelity 3d face reconstruction. In CVPR, 2020.
  38. Faceshifter: Towards high fidelity and occlusion aware face swapping. In CVPR, 2020a.
  39. Learning formation of physically-based face attributes. In CVPR, 2020b.
  40. Learning formation of physically-based face attributes. In CVPR, 2020c.
  41. 3d-aware face swapping. In CVPR, 2023.
  42. Roi tanh-polar transformer network for face parsing in the wild. IVC, 2021.
  43. Learning image-adaptive codebooks for class-agnostic image restoration. In ICCV, 2023a.
  44. Mogface: Towards a deeper appreciation on face detection. In CVPR, 2022.
  45. Damofd: Digging into backbone design on face detection. In ICLR, 2023b.
  46. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
  47. Fine-grained face swapping via regional gan inversion. In CVPR, 2023c.
  48. Normalized avatar synthesis using stylegan and perceptual refinement. In CVPR, 2021.
  49. Styleface: Towards identity-disentangled face generation on megapixels. In ECCV, 2022.
  50. Pixel codec avatars. In CVPR, 2021.
  51. Relightify: Relightable 3d faces from a single image via diffusion models. In ICCV, 2023.
  52. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
  53. Reinforced disentanglement for face swapping without skip connection. In CVPR, 2023a.
  54. Improving fairness in facial albedo estimation via visual-textual cues. In CVPR, 2023b.
  55. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
  56. Projected gans converge faster. In NeurIPS, 2021.
  57. Self-supervised monocular 3D face reconstruction by occlusion-aware multi-view geometry consistency. In ECCV, 2020.
  58. Blendface: Re-designing identity encoders for face-swapping. In CVPR, 2023.
  59. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
  60. A morphable face albedo model. In CVPR, 2020.
  61. Fml: Face model learning from videos. In CVPR, 2019.
  62. Nonlinear 3d face morphable model. In CVPR, 2018.
  63. Neural discrete representation learning. In NeurIPS, 2017.
  64. Estimating coloured 3d face models from single images: An example based approach. In ECCV, 1998.
  65. Cosface: Large margin cosine loss for deep face recognition. In CVPR, 2018.
  66. Towards real-world blind face restoration with generative facial prior. In CVPR, 2021a.
  67. Hififace: 3d shape and semantic prior guided high fidelity face swapping. In CVPR, 2021b.
  68. Self-supervised 3D face reconstruction via conditional estimation. In ICCV, 2021.
  69. Designing one unified framework for high-fidelity face reenactment and swapping. In ECCV, 2022a.
  70. Region-aware face swapping. In CVPR, 2022b.
  71. Styleswap: Style-based generator empowers robust face swapping. In ECCV, 2022c.
  72. High-fidelity facial reflectance and geometry inference from an unconstrained image. TOG, 2018.
  73. Dreamface: Progressive generation of animatable 3d faces under text guidance. In SIGGRAPH, 2023.
  74. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
  75. Towards robust blind face restoration with codebook lookup transformer. In NeurIPS, 2022.
  76. One shot face swapping on megapixels. In CVPR, 2021a.
  77. Webface260m: A benchmark unveiling the power of million-scale deep face recognition. In CVPR, 2021b.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. Monocular Identity-Conditioned Facial Reflectance Reconstruction
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets