- The paper introduces a novel unsupervised method for portrait shadow removal using the generative priors of StyleGAN2.
- It employs a progressive optimization strategy to decompose shadowed images into shadow-free and full-shadow components.
- Results demonstrate competitive performance with superior SSIM and LPIPS metrics, extending the approach to watermark and tattoo removal.
Unsupervised Portrait Shadow Removal via Generative Priors
The paper "Unsupervised Portrait Shadow Removal via Generative Priors" introduces an innovative approach to address the challenge of removing shadows from portrait images without the need for supervised training data. Leveraging the generative capabilities of pre-existing neural networks, particularly StyleGAN2, the authors present a method where the shadow removal task is interpreted as a layer decomposition problem. The central thesis posits that a shadowed portrait can be decomposed into two component images: a shadow-free image and a full-shadow image, blended together by a shadow mask.
Leveraging the rich generative priors found in StyleGAN2, the authors developed a novel progressive optimization strategy. This strategy aims to solve image decomposition problems without the necessity of paired shadow-free and shadow-present datasets. Through qualitative and quantitative assessments on an actual portrait shadow dataset, the paper demonstrates that this unsupervised method achieves results comparable to state-of-the-art supervised methods.
The paper highlights three main contributions. First, it introduces the use of StyleGAN2, which possesses deep generative priors to perform unsupervised portrait shadow removal for the first time. This model effectively addresses the unknown degradation processes inherent in shadow-ridden images, a task not achievable with existing GAN-inversion methods. Second, the authors propose a progressive optimization framework to mitigate challenges associated with shadow degradation learning from single input images. Third, they evidence that the same unsupervised methodology can be adapted to other domains such as watermark and tattoo removal from portrait images—a flexibility not possible with current supervised learning techniques.
The extensibility of the approach is noteworthy as the paper reports successes in tattoo and watermark removal as a unified framework. This speaks to the robustness of the system in handling various tasks by simply modifying the decomposition formula to account for tattoos and watermarks, using minimal changes to the existing architecture.
The experimental analysis includes comparisons with prior methods, showcasing superior performance in SSIM and LPIPS perceptual quality metrics. Moreover, despite the unsupervised nature, the method shows competent handling of complex shadow patterns which traditionally require substantial paired data for supervised learning models.
While the demonstrated versatility of the approach is promising, the paper acknowledges certain limitations regarding finely detailed aspects of portraits, such as wrinkles and accessories that are outside the training domain of StyleGAN2. Looking forward, improvements in GAN inversion techniques and expanded expressive capabilities of GAN models, particularly StyleGAN2, are anticipated to enhance outcomes in the shadow-free portrait image reconstruction.
In summary, this paper contributes significant advancements in the field of computational photography and image processing by exploiting generative priors in unsupervised learning for graphical tasks traditionally bound by supervised training data. This approach opens up possibilities for AI models to perform complex image reconstruction tasks with minimal prior data preparation, potentially democratizing high-quality portrait photography for non-professional settings.