Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Holo-Relighting: Controllable Volumetric Portrait Relighting from a Single Image (2403.09632v1)

Published 14 Mar 2024 in cs.CV

Abstract: At the core of portrait photography is the search for ideal lighting and viewpoint. The process often requires advanced knowledge in photography and an elaborate studio setup. In this work, we propose Holo-Relighting, a volumetric relighting method that is capable of synthesizing novel viewpoints, and novel lighting from a single image. Holo-Relighting leverages the pretrained 3D GAN (EG3D) to reconstruct geometry and appearance from an input portrait as a set of 3D-aware features. We design a relighting module conditioned on a given lighting to process these features, and predict a relit 3D representation in the form of a tri-plane, which can render to an arbitrary viewpoint through volume rendering. Besides viewpoint and lighting control, Holo-Relighting also takes the head pose as a condition to enable head-pose-dependent lighting effects. With these novel designs, Holo-Relighting can generate complex non-Lambertian lighting effects (e.g., specular highlights and cast shadows) without using any explicit physical lighting priors. We train Holo-Relighting with data captured with a light stage, and propose two data-rendering techniques to improve the data quality for training the volumetric relighting system. Through quantitative and qualitative experiments, we demonstrate Holo-Relighting can achieve state-of-the-arts relighting quality with better photorealism, 3D consistency and controllability.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (89)
  1. Shape, illumination, and reflectance from shading. IEEE transactions on pattern analysis and machine intelligence, 37(8):1670–1687, 2014.
  2. Deep relightable appearance models for animatable faces. ACM TOG, 40(4):1–15, 2021.
  3. A morphable model for the synthesis of 3d faces. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 157–164. 2023.
  4. Photoapp: Photorealistic appearance editing of head portraits. ACM TOG, 40(4):1–16, 2021.
  5. Large scale gan training for high fidelity natural image synthesis. In ICLR, 2018.
  6. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In CVPR, pages 5799–5809, 2021.
  7. Efficient geometry-aware 3d generative adversarial networks. In CVPR, pages 16123–16133, 2022.
  8. Rendering with style: combining traditional and neural approaches for high-quality face rendering. ACM TOG, 40(6):1–14, 2021.
  9. Learning implicit fields for generative shape modeling. In CVPR, pages 5939–5948, 2019.
  10. Intermediate layer optimization for inverse problems using deep generative models. pages 2421–2432. PMLR, 2021.
  11. Acquiring the reflectance field of a human face. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 145–156, 2000.
  12. Lumigan: Unconditional generation of relightable 3d human faces. arXiv preprint arXiv:2304.13153, 2023.
  13. Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set. In CVPRW, 2019.
  14. Gram: Generative radiance manifolds for 3d-aware image generation. In CVPR, pages 10673–10683, 2022.
  15. Learning an animatable detailed 3d face model from in-the-wild images. ACM Transactions on Graphics (ToG), 40(4):1–13, 2021.
  16. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
  17. Robin Green. Spherical harmonic lighting: The gritty details. In Archives of the game developers conference, page 4, 2003.
  18. Christopher Grey. Master lighting guide for portrait photographers. Amherst Media, 2014.
  19. Stylenerf: A style-based 3d aware generator for high-resolution image synthesis. In ICLR, 2021.
  20. The relightables: Volumetric performance capture of humans with realistic relighting. ACM TOG, 38(6):1–19, 2019.
  21. Poly Haven. Poly haven.
  22. Leveraging 2d data to learn textured 3d mesh generation. In CVPR, pages 7498–7507, 2020.
  23. Escaping plato’s cave: 3d shape from adversarial rendering. In ICCV, pages 9984–9993, 2019.
  24. Towards high fidelity face relighting with realistic shadows. In CVPR, pages 14719–14728, 2021.
  25. Face relighting with geometrically consistent shadows. In CVPR, pages 4217–4226, 2022.
  26. Dynamic 3d avatar creation from hand-held video input. ACM TOG, 34(4):1–14, 2015.
  27. Geometry-aware single-image full-body human relighting. 2022.
  28. Nerffacelighting: Implicit and disentangled face lighting representation leveraging generative prior in neural radiance fields. ACM TOG, 42(3):1–18, 2023.
  29. Learning category-specific mesh reconstruction from image collections. In ECCV, pages 371–386, 2018.
  30. Analyzing and improving the image quality of stylegan. In CVPR, pages 8110–8119, 2020.
  31. Modnet: Real-time trimap-free portrait matting via objective decomposition. In AAAI, 2022.
  32. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA, USA, 2015.
  33. Illumination-invariant face recognition with deep relit face images. In IEEE Winter Conference on Applications of Computer Vision, pages 2146–2155. IEEE, 2019.
  34. Intrinsic face image decomposition with human face priors. In ECCV, pages 218–233. Springer, 2014.
  35. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. Rendering Techniques, 2007(9):10, 2007.
  36. Lightpainter: Interactive portrait relighting with freehand scribble. In CVPR, pages 195–205, 2023.
  37. Deep relightable textures: volumetric performance capture with neural rendering. ACM TOG, 39(6):1–21, 2020.
  38. Pulse: Self-supervised photo upsampling via latent space exploration of generative models. In CVPR, pages 2437–2445, 2020.
  39. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021.
  40. Learning physics-guided face relighting under directional light. In CVPR, pages 5124–5133, 2020.
  41. Hologan: Unsupervised learning of 3d representations from natural images. In ICCV, pages 7588–7597, 2019.
  42. Giraffe: Representing scenes as compositional generative neural feature fields. In CVPR, pages 11453–11464, 2021.
  43. Stylesdf: High-resolution 3d-consistent image and geometry generation. In CVPR, pages 13503–13513, 2022.
  44. A shading-guided generative implicit model for shape-accurate 3d-aware image synthesis. NeurIPS, 34:20002–20013, 2021.
  45. Total relighting: learning to relight portraits for background replacement. ACM TOG, 40(4):1–21, 2021.
  46. Relightify: Relightable 3d faces from a single image via diffusion models. In ICCV, 2023.
  47. Post-production facial performance relighting using reflectance transfer. ACM TOG, 26(3):52–es, 2007.
  48. Facelit: Neural 3d relightable faces. In CVPR, pages 8619–8628, 2023.
  49. Vorf: Volumetric relightable faces. 2022.
  50. Encoding in style: a stylegan encoder for image-to-image translation. In CVPR, pages 2287–2296, 2021.
  51. Pivotal tuning for latent-based editing of real images. ACM TOG, 42(1):1–13, 2022.
  52. U-net: Convolutional networks for biomedical image segmentation. pages 234–241. Springer, 2015.
  53. Complete Self-instructing Library of Practical Photography: Negative retouching; etching and modeling; encyclopedic index. American school of art and photography, 1909.
  54. Graf: Generative radiance fields for 3d-aware image synthesis. NeurIPS, 33:20154–20166, 2020.
  55. Realistic inverse lighting from a single 2d image of a face, taken under unknown and complex lighting. In 2015 11th IEEE international conference and workshops on automatic face and gesture recognition (FG), pages 1–8. IEEE, 2015.
  56. The quotient image: Class-based re-rendering and recognition with varying illuminations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(2):129–139, 2001.
  57. Style transfer for headshot portraits. ACM TOG, 33(4):1–14, 2014.
  58. Portrait lighting transfer using a mass transport approach. ACM TOG, 36(4):1, 2017.
  59. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  60. Single image portrait relighting. ACM TOG, 38(4):79–1, 2019.
  61. Nelf: Neural light-transport field for portrait view synthesis and relighting. arXiv preprint arXiv:2107.12351, 2021.
  62. Volux-gan: A generative model for 3d face synthesis with hdri relighting. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1–9, 2022.
  63. Pie: Portrait image embedding for semantic control. ACM TOG, 39(6):1–14, 2020a.
  64. Stylerig: Rigging stylegan for 3d control over portrait images. In CVPR. IEEE, 2020b.
  65. Monocular reconstruction of neural face reflectance fields. In CVPR, pages 4791–4800, 2021.
  66. Towards real-world blind face restoration with generative facial prior. In CVPR, pages 9168–9178, 2021.
  67. Free-view face relighting using a hybrid parametric neural model on a small-olat dataset. IJCV, 131(4):1002–1021, 2023a.
  68. Sunstage: Portrait reconstruction and relighting using the sun as a light stage. In CVPR, pages 20792–20802, 2023b.
  69. Zhou Wang. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004.
  70. Single image portrait relighting via explicit multiple reflectance channel modeling. ACM TOG, 39(6):1–13, 2020.
  71. Performance relighting and reflectance transformation with time-multiplexed illumination. ACM TOG, 24(3):756–764, 2005.
  72. A light cnn for deep face representation with noisy labels. IEEE Transactions on Information Forensics and Security, 13(11):2884–2896, 2018.
  73. Gan inversion: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3121–3138, 2022.
  74. High-fidelity 3d gan inversion by pseudo-multi-view optimization. In CVPR, pages 321–331, 2023.
  75. Pastiche master: Exemplar-based high-resolution portrait style transfer. In CVPR, pages 7693–7702, 2022a.
  76. Vtoonify: Controllable high-resolution portrait video style transfer. ACM TOG, 41(6):1–15, 2022b.
  77. Gan prior embedded network for blind face restoration in the wild. In CVPR, pages 672–681, 2021.
  78. Learning to relight portrait images via a virtual light stage and synthetic-to-real adaptation. ACM TOG, 2022.
  79. 3d gan inversion with facial symmetry prior. In CVPR, pages 342–351, 2023.
  80. Free-form image inpainting with gated convolution. In ICCV, pages 4471–4480, 2019.
  81. Make encoder great again in 3d gan inversion through geometry and occlusion-aware encoding. arXiv preprint arXiv:2303.12326, 2023.
  82. A feature-enriched completely blind image quality evaluator. IEEE Transactions on Image Processing, 24(8):2579–2591, 2015.
  83. Neural video portrait relighting in real-time via consistency modeling. In ICCV, pages 802–812, 2021a.
  84. Adding conditional control to text-to-image diffusion models. In ICCV, pages 3836–3847, 2023.
  85. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, pages 586–595, 2018.
  86. Portrait shadow manipulation. ACM TOG, 39(4):78–1, 2020.
  87. Neural light transport for relighting and view synthesis. ACM TOG, 40(1):1–17, 2021b.
  88. Deep single-image portrait relighting. In ICCV, pages 7194–7202, 2019.
  89. Cips-3d: A 3d-aware generator of gans based on conditionally-independent pixel synthesis. arXiv preprint arXiv:2110.09788, 2021.
Citations (9)
View on Semantic Scholar

Summary

  • The paper introduces a volumetric relighting method that controls lighting, viewpoint, and pose using a single portrait image.
  • It employs a pretrained EG3D model and novel shading transfer techniques to render photorealistic specular highlights and shadows.
  • Extensive experiments demonstrate enhanced realism and versatility, setting a new benchmark in computational portrait photography.

Holo-Relighting: Advanced Volumetric Portrait Relighting from Single Images

Introduction to Holo-Relighting

In the pursuit of flexibility and accessibility in portrait photography, the ability to manipulate lighting and viewpoint after capturing an image presents a compelling advantage. This paper introduces Holo-Relighting, a method poised to address the challenges associated with recreating the lighting of a portrait. Holo-Relighting utilizes a pretrained 3D GAN, specifically EG3D, to remodel the geometry and appearance from a supplied portrait, generating a tri-plane-based 3D representation that is sensitive to lighting conditions. This approach enables intricate control over lighting effects, camera viewpoints, and head poses from merely a single image.

Key Contributions

  • The introduction of a volumetric relighting method that empowers users with granular control over lighting, viewpoint, and head pose, directly from a single portrait image.
  • An innovative relighting module designed to render complex shading effects including specular highlights and cast shadows, without resorting to physical lighting assumptions.
  • Two novel data-rendering techniques aimed at leveraging light stage capture data to enhance the training of volumetric relighting systems.

Volumetric Relighting Approach

The core innovation of Holo-Relighting lies in its ability to synthesize relit portraits with remarkable realism and consistency across different views, utilizing a volumetric approach. By reconstructing the subject's 3D geometry and appearance through the inversion of a pretrained EG3D model, Holo-Relighting achieves a level of detail and control previously unattainable with 2D methods or simplified 3D assumptions. This volumetric technique allows for the rendering of novel views under variable lighting conditions and head poses, a significant advancement in the field of computational photography.

Relighting Without Physical Assumptions

One of the standout features of Holo-Relighting is its departure from explicit physical lighting models. Instead, the method relies on a learned neural network to intuitively understand and replicate lighting effects. This approach not only simplifies the model but also enhances its ability to generate non-Lambertian effects such as cast shadows and specular highlights, all without predefined physical constraints. This represents a substantial leap forward, enabling more nuanced and naturalistic lighting manipulations.

Enhanced Data-Rendering Techniques

Training a volumetric relighting system requires high-quality data that accurately represents a wide range of lighting conditions. Holo-Relighting introduces two innovative techniques to improve the utilization of light stage capture data: multi-view regularization and shading transfer. These techniques ensure that the 3D representation captures precise geometry and aligns with the target illumination, thus, significantly enhancing the system's ability to render realistic shading effects.

Experimental Validation

Extensive quantitative and qualitative evaluations demonstrate the superior performance of Holo-Relighting over existing methods. The system excels in rendering quality, offering unparalleled photorealism, 3D consistency, and controllability, even when compared to state-of-the-art relighting approaches. Through various experiments, Holo-Relighting showcases its robustness to diverse subjects and lighting conditions, affirming its potential for practical photographic applications.

Implications and Future Directions

Holo-Relighting sets a new benchmark for portrait relighting, offering both theoretical and practical advancements in the field. Its ability to bypass explicit physical lighting models while achieving high-quality, controllable results opens new avenues for research into implicit relighting techniques. Prospects for future development include exploring the integration of Holo-Relighting with dynamic scenes and extending its application beyond portrait photography to achieve broader impacts on visual media production.

Conclusion

Holo-Relighting emerges as a pioneering method in volumetric portrait relighting, pushing the boundaries of what is achievable with a single image input. Its innovative approach, coupled with the enhanced data-rendering techniques, positions it as a transformative tool for photographers, artists, and researchers alike. As the field of computationally assisted photography continues to evolve, Holo-Relighting stands out as a key contribution, paving the way for even more creative and flexible photographic techniques.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets