ProRes: Exploring Degradation-aware Visual Prompt for Universal Image Restoration (2306.13653v1)

Abstract: Image restoration aims to reconstruct degraded images, e.g., denoising or deblurring. Existing works focus on designing task-specific methods and there are inadequate attempts at universal methods. However, simply unifying multiple tasks into one universal architecture suffers from uncontrollable and undesired predictions. To address those issues, we explore prompt learning in universal architectures for image restoration tasks. In this paper, we present Degradation-aware Visual Prompts, which encode various types of image degradation, e.g., noise and blur, into unified visual prompts. These degradation-aware prompts provide control over image processing and allow weighted combinations for customized image restoration. We then leverage degradation-aware visual prompts to establish a controllable and universal model for image restoration, called ProRes, which is applicable to an extensive range of image restoration tasks. ProRes leverages the vanilla Vision Transformer (ViT) without any task-specific designs. Furthermore, the pre-trained ProRes can easily adapt to new tasks through efficient prompt tuning with only a few images. Without bells and whistles, ProRes achieves competitive performance compared to task-specific methods and experiments can demonstrate its ability for controllable restoration and adaptation for new tasks. The code and models will be released in \url{https://github.com/leonmakise/ProRes}.

ProRes: Exploring Degradation-aware Visual Prompt for Universal Image Restoration

The paper presents a novel approach to universal image restoration through a technique called ProRes, which leverages degradation-aware visual prompts. The significance of image restoration as a fundamental task within computer vision arises from its applicability in enhancing degraded images afflicted by factors such as low light, noise, blur, and rain. Traditionally, image restoration methods have been designed with task-specific architectures, which although effective, do not generalize well across different types of degradation. ProRes seeks to address these limitations by offering a versatile framework capable of handling various restoration tasks through a unified model. The methodology is centered around the integration of degradation-aware visual prompts that facilitate a controllable approach to image restoration.

Context and Motivation

Traditional image restoration tasks have typically utilized deep learning techniques with highly specialized architectures tailored for particular degradation issues—such as denoising, deraining, and deblurring. These specialized solutions lack transferability across various restoration problems, necessitating unique model designs for each task. The introduction of ProRes seeks to overcome this inefficiency by proposing a universal model capable of managing multiple image degradation challenges without task-specific modifications.

Degradation-aware Visual Prompts

The core innovation of ProRes lies in degradation-aware visual prompts, which act as parametric identifiers for distinct types of image degradation. By encoding different restorations within their respective prompts, ProRes allows for precise control over the restoration process. This system facilitates both the selection of single-task prompts and the combination of multiple prompts in a weighted manner, thereby enabling the handling of images subjected to multiple forms of degradation.

Methodology

ProRes employs a vanilla Vision Transformer (ViT) architecture without the need for bespoke designs for each restoration task. It operates by adding task-specific visual prompts directly to the input images. These prompts aid ViT in discerning the type and degree of degradation, guiding the restoration process accordingly. Importantly, the visual prompts are pre-trained with a lightweight model to ensure effective initialization before they are fine-tuned with the larger ViT architecture. One of the substantial requirements fulfilled by ProRes is the ability to adapt seamlessly to new tasks or datasets through prompt tuning, which updates only the prompts rather than the entire model.

Empirical Evaluation

The experiments conducted illustrate the competitive performance of ProRes across a host of image restoration tasks including denoising, deraining, low-light enhancement, and deblurring. The numerical results indicate that ProRes is on par with established task-specific methods, reinforcing its capability as a versatile, scalable solution for low-level vision tasks. Additionally, visualizations demonstrate the strong control ProRes offers over restoration outputs by adjusting the combination and weighting of prompts. Notably, ProRes effectively adapted to new datasets and tasks, indicating its robust transferability via prompt tuning.

Implications and Future Work

ProRes is indicative of the growing trend towards universal models in computer vision that aim to diminish the limitations associated with task specificity. By introducing degradation-aware visual prompts, the researchers set a foundation for future exploration into more sophisticated universal frameworks that could handle a broader range of vision tasks more efficiently. Further investigation could seek to enhance the control and granularity with which restoration tasks are executed by developing more sophisticated prompts or incorporating additional model architectures.

The paper provides valuable insights into the potential amalgamation of prompt learning and universal modeling to achieve both breadth and specificity in image restoration. It stands as a potential benchmark for further research efforts aimed at enhancing the scalability and adaptability of image restoration models.