Towards Vivid and Diverse Image Colorization with Generative Color Prior (2108.08826v2)

Abstract: Colorization has attracted increasing interest in recent years. Classic reference-based methods usually rely on external color images for plausible results. A large image database or online search engine is inevitably required for retrieving such exemplars. Recent deep-learning-based methods could automatically colorize images at a low cost. However, unsatisfactory artifacts and incoherent colors are always accompanied. In this work, we propose GCP-Colorization that leverages the rich and diverse color priors encapsulated in a pretrained Generative Adversarial Networks (GAN) for automatic colorization. Specifically, we first "retrieve" matched features (similar to exemplars) via a GAN encoder and then incorporate these features into the colorization process with feature modulations. Thanks to the powerful generative color prior (GCP) and delicate designs, our GCP-Colorization could produce vivid colors with a single forward pass. Moreover, it is highly convenient to obtain diverse results by modifying GAN latent codes. GCP-Colorization also inherits the merit of interpretable controls of GANs and could attain controllable and smooth transitions by walking through GAN latent space. Extensive experiments and user studies demonstrate that GCP-Colorization achieves superior performance than previous works. Codes are available at https://github.com/ToTheBeginning/GCP-Colorization.

• The paper introduces GCP-Colorization, leveraging generative color prior from pretrained GANs to achieve vivid and diverse image colorization.
• The method integrates GAN-derived features into the colorization network using SPADE layers and employs spatial alignment for coherence.
• Experiments demonstrate improved vividness and diversity over state-of-the-art methods, showcasing the utility of generative color priors for image colorization.

Overview of "Towards Vivid and Diverse Image Colorization with Generative Color Prior"

This paper presents a novel approach to the image colorization problem by utilizing the generative color prior (GCP) encapsulated in pretrained Generative Adversarial Networks (GANs). The proposed GCP-Colorization method aims to address the challenges of automatic image colorization, notably the production of vivid and diverse color outputs, without relying on external references or manual input.

Key Contributions

The GCP-Colorization framework introduces several key innovations:

1. Generative Color Prior Utilization: It leverages the rich color information inherent in pretrained GAN models to guide the colorization process. This approach circumvents the need for explicit retrieval of exemplar images, which is common in traditional reference-based methods.
2. Feature Modulation: The method integrates GAN-derived features into the colorization network through spatially-adaptive denormalization (SPADE) layers. This facilitates vivid and coherent color generation across the image.
3. Diverse and Controllable Colorization: By manipulating GAN latent codes, the framework can produce a variety of plausible colorization outcomes, offering smooth transitions and adjustable results.

Methodology

The proposed method comprises several components:

• GAN Encoder: This maps grayscale images to a latent code space that is used by the generator to produce an inversion image and related intermediate features.
• Colorization Network: This uses a combination of downsampling, residual, and upsampling blocks to predict the missing color channels, enhanced by SPADE modulation using aligned GAN features.
• Spatial Alignment: A non-local operation dynamically aligns GAN features with the input grayscale image, ensuring coherence and mitigating spatial discrepancy between them.

Results and Evaluation

Extensive experiments demonstrate the efficacy of GCP-Colorization in generating more vivid and diverse colorization results compared to state-of-the-art methods like CIC, ChromaGAN, DeOldify, and InstColor. Quantitative metrics show improvements in FID and colorfulness scores, while user studies indicate a preference for the results produced by the proposed framework.

Implications and Future Directions

The approach exemplifies the powerful applicability of GAN-generated priors in enhancing automatic image-to-image translation tasks. It suggests a promising direction for developing more sophisticated generative models that can provide rich, diverse, and controllable outputs across various domains. Future work could explore further refinements in GAN inversion techniques and expand the framework to other colorization challenges where GANs may have limited coverage.

Overall, the paper offers an insightful contribution to the ongoing development of image colorization, integrating generative modeling techniques to advance the task beyond traditional limitations.