Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Robust Unsupervised StyleGAN Image Restoration (2302.06733v2)

Published 13 Feb 2023 in cs.CV

Abstract: GAN-based image restoration inverts the generative process to repair images corrupted by known degradations. Existing unsupervised methods must be carefully tuned for each task and degradation level. In this work, we make StyleGAN image restoration robust: a single set of hyperparameters works across a wide range of degradation levels. This makes it possible to handle combinations of several degradations, without the need to retune. Our proposed approach relies on a 3-phase progressive latent space extension and a conservative optimizer, which avoids the need for any additional regularization terms. Extensive experiments demonstrate robustness on inpainting, upsampling, denoising, and deartifacting at varying degradations levels, outperforming other StyleGAN-based inversion techniques. Our approach also favorably compares to diffusion-based restoration by yielding much more realistic inversion results. Code is available at https://lvsn.github.io/RobustUnsupervised/.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (66)
  1. Diffjpeg. https://github.com/mlomnitz/DiffJPEG.
  2. Libjpeg. https://libjpeg-turbo.org.
  3. Image2StyleGAN: How to embed images into the StyleGAN latent space? In Int. Conf. Comput. Vis., 2019.
  4. Image2stylegan++: How to edit the embedded images? In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
  5. Brecahad: a dataset for breast cancer histopathological annotation and diagnosis. BMC research notes, 12(1):1–3, 2019.
  6. Compressed sensing using generative models. In Int. Conf. Mach. Learn., 2017.
  7. Large scale GAN training for high fidelity natural image synthesis. In Int. Conf. Learn. Represent., 2019.
  8. Any-resolution training for high-resolution image synthesis. In Eur. Conf. Comput. Vis., 2022.
  9. Pre-trained image processing transformer. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
  10. Proxiqa: A proxy approach to perceptual optimization of learned image compression. IEEE Trans. Image Process., 30:360–373, 2020.
  11. Stargan v2: Diverse image synthesis for multiple domains. In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
  12. Alex Clark. Pillow (pil fork) documentation, 2015.
  13. Stylegan-induced data-driven regularization for inverse problems. In Int. Conf. Acous. Sp. Sig. Proc., 2022.
  14. Score-guided intermediate layer optimization: Fast Langevin mixing for inverse problems. In Int. Conf. Mach. Learn., 2022.
  15. Intermediate layer optimization for inverse problems using deep generative models. In Int. Conf. Mach. Learn., 2021.
  16. Arcface: Additive angular margin loss for deep face recognition. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
  17. Diffusion models beat gans on image synthesis. In Adv. Neural Inform. Process. Syst., 2021.
  18. Image inpainting: A review. Neural Processing Letters, 51:2007–2028, 2019.
  19. Brief review of image denoising techniques. Visual Computing for Industry, Biomedicine and Art, 2, 2019.
  20. Generative adversarial nets. In Adv. Neural Inform. Process. Syst., 2014.
  21. Image processing using multi-code gan prior. In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
  22. Vqfr: Blind face restoration with vector-quantized dictionary and parallel decoder. In Eur. Conf. Comput. Vis., 2022.
  23. Ganspace: Discovering interpretable gan controls. In Adv. Neural Inform. Process. Syst., 2020.
  24. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Adv. Neural Inform. Process. Syst., 2017.
  25. Image-adaptive gan based reconstruction. In Ass. Adv. Artif. Intel., 2020.
  26. Gan inversion for out-of-range images with geometric transformations. In Int. Conf. Comput. Vis., 2021.
  27. Scaling up GANs for text-to-image synthesis. In IEEE Conf. Comput. Vis. Pattern Recog., 2023.
  28. Training generative adversarial networks with limited data. In Adv. Neural Inform. Process. Syst., 2020.
  29. Alias-free generative adversarial networks. In Adv. Neural Inform. Process. Syst., 2021.
  30. A style-based generator architecture for generative adversarial networks. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
  31. Analyzing and improving the image quality of StyleGAN. In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
  32. Denoising diffusion restoration models. In Adv. Neural Inform. Process. Syst., 2022.
  33. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  34. Faceformer: Scale-aware blind face restoration with transformers. ArXiv, abs/2207.09790, 2022.
  35. From beginner to master: A survey for deep learning-based single-image super-resolution. arXiv preprint arXiv:2109.14335, 2021.
  36. Enhanced blind face restoration with multi-exemplar images and adaptive spatial feature fusion. In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
  37. Swinir: Image restoration using swin transformer. In Int. Conf. Comput. Vis., 2021.
  38. Time-travel rephotography. ACM Trans. Graph., 40(6):1–12, 2021.
  39. Bayesian image reconstruction using deep generative models. arXiv preprint arXiv:2012.04567, 2020.
  40. Pulse: Self-supervised photo upsampling via latent space exploration of generative models. In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
  41. NVlabs. Stylegan2 ada - pytorch. https://github.com/NVlabs/stylegan2-ada-pytorch.
  42. Exploiting deep generative prior for versatile image restoration and manipulation. IEEE Trans. Pattern Anal. Mach. Intell., 2021.
  43. Styleclip: Text-driven manipulation of stylegan imagery. In Int. Conf. Comput. Vis., 2021.
  44. Overparameterization improves StyleGAN inversion. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
  45. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
  46. Pivotal tuning for latent-based editing of real images. ACM Trans. Graph., 42(1), Feb. 2023.
  47. High-resolution image synthesis with latent diffusion models. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
  48. Palette: Image-to-image diffusion models. ACM Trans. Graph., pages 1–10, 2022.
  49. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487, 2022.
  50. StyleGAN-XL: Scaling stylegan to large diverse datasets. ACM Trans. Graph., 1, 2022.
  51. Interfacegan: Interpreting the disentangled face representation learned by gans. IEEE Trans. Pattern Anal. Mach. Intell., 2020.
  52. Jpeg-resistant adversarial images. In NeurIPS Workshop Mach. Learn. Comp. Sec., 2017.
  53. Deep unsupervised learning using nonequilibrium thermodynamics. In Int. Conf. Mach. Learn., 2015.
  54. Maxim: Multi-axis MLP for image processing. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
  55. Stylegan2 distillation for feed-forward image manipulation. In Eur. Conf. Comput. Vis., 2020.
  56. Towards real-world blind face restoration with generative facial prior. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
  57. Deep learning for image super-resolution: A survey. IEEE Trans. Pattern Anal. Mach. Intell., 43(10):3365–3387, 2020.
  58. Machine Learning Refined: Foundations, Algorithms, and Applications. Cambridge University Press, 2016.
  59. GAN inversion: A survey. IEEE Trans. Pattern Anal. Mach. Intell., 2022.
  60. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015.
  61. Learning enriched features for real image restoration and enhancement. In Eur. Conf. Comput. Vis., 2020.
  62. Multi-stage progressive image restoration. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
  63. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
  64. Towards robust blind face restoration with codebook lookup transformer, 2022.
  65. Blind face restoration via integrating face shape and generative priors. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
  66. Improved stylegan embedding: Where are the good latents? arXiv preprint arXiv:2012.09036, 2020.
Citations (15)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now