Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

JoIN: Joint GANs Inversion for Intrinsic Image Decomposition (2305.11321v2)

Published 18 May 2023 in cs.CV

Abstract: In this work, we propose to solve ill-posed inverse imaging problems using a bank of Generative Adversarial Networks (GAN) as a prior and apply our method to the case of Intrinsic Image Decomposition for faces and materials. Our method builds on the demonstrated success of GANs to capture complex image distributions. At the core of our approach is the idea that the latent space of a GAN is a well-suited optimization domain to solve inverse problems. Given an input image, we propose to jointly inverse the latent codes of a set of GANs and combine their outputs to reproduce the input. Contrary to most GAN inversion methods which are limited to inverting only a single GAN, we demonstrate that it is possible to maintain distribution priors while inverting several GANs jointly. We show that our approach is modular, allowing various forward imaging models, and that it can successfully decompose both synthetic and real images.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (87)
  1. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4432–4441, 2019.
  2. Image2stylegan++: How to edit the embedded images? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8296–8305, 2020.
  3. Two-shot svbrdf capture for stationary materials. ACM Trans. Graph., 34(4), 2015.
  4. Restyle: A residual-based stylegan encoder via iterative refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6711–6720, 2021.
  5. Shape, illumination, and reflectance from shading. IEEE Trans. Pattern Anal. Mach. Intell., 37(8):1670–1687, 2015.
  6. Semantic photo manipulation with a generative image prior. ACM Transactions on Graphics (TOG), 38:1 – 11, 2019a.
  7. Seeing what a gan cannot generate. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4502–4511, 2019b.
  8. Seeing what a gan cannot generate. ICCV, pages 4501–4510, 2019c.
  9. Stylegan knows normal, depth, albedo, and more. ArXiv, abs/2306.00987, 2023.
  10. Blender Online Community. Blender - a 3D modelling and rendering package. Blender Foundation, Blender Institute, Amsterdam, 2023.
  11. Interactive intrinsic video editing. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2014), 33(6), 2014.
  12. Compressed sensing using generative models. In International Conference on Machine Learning, 2017.
  13. User Assisted Intrinsic Images. ACM Transactions on Graphics, 28(5):130:1–10, 2009.
  14. Neural photo editing with introspective adversarial networks. arXiv preprint arXiv:1609.07093, 2016.
  15. Using latent space regression to analyze and leverage compositionality in gans. ICLR, 2021a.
  16. Ensembling with deep generative views. In CVPR, 2021b.
  17. Inverting the generator of a generative adversarial network. IEEE Transactions on Neural Networks and Learning Systems, 30:1967–1974, 2019.
  18. Signet: Intrinsic image decomposition by a semantic and invariant gradient driven network for indoor scenes. In European Conference of Computer Vision Workshop CV4Metaverse 2022 (ECCV)., 2022.
  19. Single-image svbrdf capture with a rendering-aware deep network. ACM Trans. Graph., 37(4), 2018.
  20. Flexible svbrdf capture with a multi-image deep network. Computer Graphics Forum, 38(4):1–13, 2019.
  21. Practical face reconstruction via differentiable ray tracing. In Computer Graphics Forum, pages 153–164. Wiley Online Library, 2021a.
  22. Towards high fidelity monocular face reconstruction with rich reflectance using self-supervised learning and ray tracing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12819–12829, 2021b.
  23. S2f2: Self-supervised high fidelity face reconstruction from monocular image. arXiv preprint arXiv:2203.07732, 2022.
  24. Multi-view intrinsic images of outdoors scenes with an application to relighting. ACM Transactions on Graphics (TOG), 34(5), 2015.
  25. Near perfect gan inversion. ArXiv, abs/2202.11833, 2022.
  26. Outcast: Single image relighting with cast shadows. Computer Graphics Forum, 43, 2022.
  27. Image processing using multi-code gan prior. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3009–3018, 2019.
  28. Materialgan: Reflectance capture using a generative svbrdf model. ACM Trans. Graph., 39(6), 2020.
  29. Phase retrieval under a generative prior. ArXiv, abs/1807.04261, 2018.
  30. Berthold K.P. Horn. Determining lightness from an image. Computer Graphics and Image Processing, 3(4):277–299, 1974.
  31. Towards high fidelity face relighting with realistic shadows. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14719–14728, 2021.
  32. Transforming and projecting images into class-conditional generative networks. In European Conference on Computer Vision, pages 17–34. Springer, 2020.
  33. Alternating phase projected gradient descent with generative priors for solving compressive phase retrieval. ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7705–7709, 2019.
  34. Progressive growing of gans for improved quality, stability, and variation. In International Conference on Learning Representations, 2018.
  35. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4401–4410, 2019a.
  36. A style-based generator architecture for generative adversarial networks, 2019b.
  37. Analyzing and improving the image quality of StyleGAN. In Proc. CVPR, 2020a.
  38. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110–8119, 2020b.
  39. E-lpips: Robust perceptual image similarity via random transformation ensembles. ArXiv, abs/1906.03973, 2019.
  40. Coherent intrinsic images from photo collections. ACM Transactions on Graphics (TOG), 31(6), 2012.
  41. Rich intrinsic image decomposition of outdoor scenes from multiple views. IEEE Transactions on Visualization and Computer Graphics, 19(2):210 – 224, 2013. presented at SIGGRAPH 2012 (Talk and Poster sessions).
  42. Lightness and retinex theory. J. Opt. Soc. Am., 61(1):1–11, 1971.
  43. Autoencoding beyond pixels using a learned similarity metric. ArXiv, abs/1512.09300, 2016.
  44. Photo-realistic single image super-resolution using a generative adversarial network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 105–114, 2016.
  45. Learning intrinsic image decomposition from watching the world. In Computer Vision and Pattern Recognition (CVPR), 2018a.
  46. Cgintrinsics: Better intrinsic image decomposition through physically-based rendering. In Proceedings of the European Conference on Computer Vision (ECCV), pages 371–387, 2018b.
  47. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2475–2484, 2020.
  48. Precise recovery of latent vectors from generative adversarial networks. ArXiv, abs/1702.04782, 2017a.
  49. Precise recovery of latent vectors from generative adversarial networks. In International Conference on Learning Representations Workshops, 2017b.
  50. Unsupervised learning for intrinsic image decomposition from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  51. Cycle encoding of a stylegan encoder for improved reconstruction and editability. Proceedings of the 30th ACM International Conference on Multimedia, 2022.
  52. Live intrinsic video. ACM Transactions on Graphics (Proceedings SIGGRAPH), 35(4), 2016.
  53. Real-time global illumination decomposition of videos. In ACM Transactions on Graphics, 2021.
  54. Interestyle: Encoding an interest region for robust stylegan inversion. In ECCV, 2022.
  55. Reflectance adaptive filtering improves intrinsic image estimation. In Computer Vision and Pattern Recognition (CVPR), 2017.
  56. Hdr-gan: Hdr image reconstruction from multi-exposed ldr images with large motions. IEEE Transactions on Image Processing, 30:3885–3896, 2020.
  57. Total relighting: Learning to relight portraits for background replacement. In ACM Transactions on Graphics (Proceedings SIGGRAPH), 2021.
  58. Invertible conditional gans for image editing. arXiv preprint arXiv:1611.06355, 2016.
  59. Multi-view relighting using a geometry-aware network. ACM Transactions on Graphics, 2019.
  60. Free-viewpoint indoor neural relighting from multi-view stereo. ACM Transactions on Graphics, 2021.
  61. Adversarial latent autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14104–14113, 2020.
  62. Photographic tone reproduction for digital images. ACM Transactions on Graphics, 21, 2002.
  63. Encoding in style: a stylegan encoder for image-to-image translation. CVPR, 2021a.
  64. Encoding in style: a stylegan encoder for image-to-image translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2287–2296, 2021b.
  65. Pivotal tuning for latent-based editing of real images. ACM Transactions on Graphics (TOG), 2021.
  66. Solving linear inverse problems using gan priors: An algorithm with provable guarantees. 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4609–4613, 2018a.
  67. Solving linear inverse problems using gan priors: An algorithm with provable guarantees. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4609–4613. IEEE, 2018b.
  68. Closed-form factorization of latent semantics in gans. In CVPR, 2021.
  69. Recovering intrinsic images from a single image. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(9):1459–1472, 2005.
  70. Recovering intrinsic images from a single image. In Advances in neural information processing systems, pages 1367–1374, 2003.
  71. Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG), 40(4):1–14, 2021.
  72. High-fidelity gan inversion for image attribute editing. arxiv:2109.06590, 2021.
  73. A simple baseline for stylegan inversion. arXiv preprint arXiv:2104.07661, 2021.
  74. Y. Weiss. Deriving intrinsic images from image sequences. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, pages 68–75 vol.2, 2001.
  75. Gan inversion: A survey. arXiv preprint arXiv:2101.05278, 2021.
  76. A style-based gan encoder for high fidelity reconstruction of images and videos. European conference on computer vision, 2022.
  77. Intrinsic video and applications. ACM Transactions on Graphics (SIGGRAPH 2014), 33(4), 2014.
  78. Semantic image inpainting with deep generative models. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6882–6890, 2016.
  79. Learning to relight portrait images via a virtual light stage and synthetic-to-real adaptation. ACM Transactions on Graphics (TOG), 2022.
  80. Inverserendernet: Learning single image inverse rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  81. Ye Yu and William A. P. Smith. Outdoor inverse rendering from a single image using multiview self-supervision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021. to appear.
  82. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
  83. Learning data-driven reflectance priors for intrinsic image decomposition. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015.
  84. Adversarial single-image svbrdf estimation with hybrid training. Computer Graphics Forum, 2021.
  85. Look-ahead training with learned reflectance loss for single-image svbrdf estimation. ACM Transactions on Graphics, 41(6), 2022.
  86. Generative visual manipulation on the natural image manifold. In ECCV, 2016.
  87. In-domain gan inversion for real image editing. In European conference on computer vision, pages 592–608. Springer, 2020.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now