Boosting Image Restoration via Priors from Pre-trained Models (2403.06793v2)
Abstract: Pre-trained models with large-scale training data, such as CLIP and Stable Diffusion, have demonstrated remarkable performance in various high-level computer vision tasks such as image understanding and generation from language descriptions. Yet, their potential for low-level tasks such as image restoration remains relatively unexplored. In this paper, we explore such models to enhance image restoration. As off-the-shelf features (OSF) from pre-trained models do not directly serve image restoration, we propose to learn an additional lightweight module called Pre-Train-Guided Refinement Module (PTG-RM) to refine restoration results of a target restoration network with OSF. PTG-RM consists of two components, Pre-Train-Guided Spatial-Varying Enhancement (PTG-SVE), and Pre-Train-Guided Channel-Spatial Attention (PTG-CSA). PTG-SVE enables optimal short- and long-range neural operations, while PTG-CSA enhances spatial-channel attention for restoration-related learning. Extensive experiments demonstrate that PTG-RM, with its compact size ($<$1M parameters), effectively enhances restoration performance of various models across different tasks, including low-light enhancement, deraining, deblurring, and denoising.
- Semantic segmentation guided real-world super-resolution. In WACV, 2022.
- A high-quality denoising dataset for smartphone cameras. In CVPR, 2018.
- Defocus deblurring using dual-pixel data. In ECCV, 2020.
- Blended diffusion for text-driven editing of natural images. In CVPR, 2022.
- Learning to see in the dark. In CVPR, 2018.
- Zero-shot out-of-distribution detection based on the pre-trained model clip. In AAAI, 2022.
- Rich Franzen. Kodak lossless true color image suite. http://r0k.us/graphics/kodak/, 1999. Online accessed 24 Oct 2021.
- Removing rain from single images via a deep detail network. In CVPR, 2017.
- Zero-reference deep curve estimation for low-light image enhancement. In CVPR, 2020.
- Single image super-resolution from transformed self-exemplars. In CVPR, 2015.
- Enlightengan: Deep light enhancement without paired supervision. TIP, 2021.
- Darkvisionnet: Low-light imaging via rgb-nir fusion with deep inconsistency prior. In AAAI, 2022.
- Edge-based defocus blur estimation with adaptive scale selection. TIP, 2017.
- Image reconstruction with predictive filter flow. arXiv preprint arXiv:1811.11482, 2018.
- Iterative filter adaptive network for single image defocus deblurring. In CVPR, 2021.
- Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022a.
- Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint, 2023a.
- Close the loop: a unified bottom-up and top-down paradigm for joint image deraining and segmentation. In AAAI, 2022b.
- Efficient and explicit modelling of image hierarchies for image restoration. In CVPR, 2023b.
- When image denoising meets high-level vision tasks: A deep learning approach. In IJCAI, 2018.
- Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement. In CVPR, 2021.
- Toward fast, flexible, and robust low-light image enhancement. In CVPR, 2022.
- A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, 2001.
- Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR, 2017.
- Styleclip: Text-driven manipulation of stylegan imagery. In ICCV, 2021.
- Spatially-adaptive image restoration using distortion-guided networks. In ICCV, 2021.
- Learning transferable visual models from natural language supervision. In ICML, 2021.
- Real-world blur dataset for learning and benchmarking deblurring algorithms. In ECCV, 2020.
- High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
- Human-aware motion deblurring. In ICCV, 2019.
- Just noticeable defocus blur detection and estimation. In CVPR, 2015.
- Purifying low-light images via near-infrared enlightened image. TMM, 2022.
- Exploring clip for assessing the look and feel of images. In AAAI, 2023a.
- Seeing dynamic scene in the dark: A high-quality video dataset with mechatronic alignment. In ICCV, 2021.
- Ultra-high-definition low-light image enhancement: A benchmark and transformer-based method. In AAAI, 2023b.
- Recovering realistic texture in image super-resolution by deep spatial feature transform. In CVPR, 2018.
- Uformer: A general u-shaped transformer for image restoration. In CVPR, 2022a.
- Blind2unblind: Self-supervised image denoising with visible blind spots. In CVPR, 2022b.
- Cris: Clip-driven referring image segmentation. In CVPR, 2022c.
- Uretinex-net: Retinex-based deep unfolding network for low-light image enhancement. In CVPR, 2022.
- Learning semantic-aware knowledge guidance for low-light image enhancement. In CVPR, 2023.
- Snr-aware low-light image enhancement. In CVPR, 2022a.
- Pvdd: A practical video denoising dataset with real-world dynamic scenes. arXiv preprint, 2022b.
- General adversarial defense against black-box attacks via pixel level and feature level distribution alignments. arXiv preprint, 2022c.
- Deep parametric 3d filters for joint video denoising and illumination enhancement in video super resolution. In AAAI, 2023a.
- Low-light image enhancement via structure modeling and guidance. In CVPR, 2023b.
- Ulip: Learning unified representation of language, image and point cloud for 3d understanding. In CVPR, 2023.
- Deep joint rain detection and removal from a single image. In CVPR, 2017.
- Sparse gradient regularized deep Retinex network for robust low-light image enhancement. TIP, 2021.
- K3dn: Disparity-aware kernel estimation for dual-pixel defocus deblurring. In CVPR, 2023.
- Multi-stage progressive image restoration. In CVPR, 2021.
- Restormer: Efficient transformer for high-resolution image restoration. In CVPR, 2022.
- Lit: Zero-shot transfer with locked-image text tuning. In CVPR, 2022.
- Density-aware single image de-raining using a multi-stream dense network. In CVPR, 2018.
- Image de-raining using a conditional generative adversarial network. TCSVT, 2019.
- Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. TIP, 2017.
- Plug-and-play image restoration with deep denoiser prior. TPAMI, 2021.
- Color demosaicking by local directional interpolation and nonlocal adaptive thresholding. JEI, 2011.
- Pointclip: Point cloud understanding by clip. In CVPR, 2022.
- Zegclip: Towards adapting clip for zero-shot semantic segmentation. In CVPR, 2023.