Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Boosting Image Restoration via Priors from Pre-trained Models (2403.06793v2)

Published 11 Mar 2024 in cs.CV

Abstract: Pre-trained models with large-scale training data, such as CLIP and Stable Diffusion, have demonstrated remarkable performance in various high-level computer vision tasks such as image understanding and generation from language descriptions. Yet, their potential for low-level tasks such as image restoration remains relatively unexplored. In this paper, we explore such models to enhance image restoration. As off-the-shelf features (OSF) from pre-trained models do not directly serve image restoration, we propose to learn an additional lightweight module called Pre-Train-Guided Refinement Module (PTG-RM) to refine restoration results of a target restoration network with OSF. PTG-RM consists of two components, Pre-Train-Guided Spatial-Varying Enhancement (PTG-SVE), and Pre-Train-Guided Channel-Spatial Attention (PTG-CSA). PTG-SVE enables optimal short- and long-range neural operations, while PTG-CSA enhances spatial-channel attention for restoration-related learning. Extensive experiments demonstrate that PTG-RM, with its compact size ($<$1M parameters), effectively enhances restoration performance of various models across different tasks, including low-light enhancement, deraining, deblurring, and denoising.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. Semantic segmentation guided real-world super-resolution. In WACV, 2022.
  2. A high-quality denoising dataset for smartphone cameras. In CVPR, 2018.
  3. Defocus deblurring using dual-pixel data. In ECCV, 2020.
  4. Blended diffusion for text-driven editing of natural images. In CVPR, 2022.
  5. Learning to see in the dark. In CVPR, 2018.
  6. Zero-shot out-of-distribution detection based on the pre-trained model clip. In AAAI, 2022.
  7. Rich Franzen. Kodak lossless true color image suite. http://r0k.us/graphics/kodak/, 1999. Online accessed 24 Oct 2021.
  8. Removing rain from single images via a deep detail network. In CVPR, 2017.
  9. Zero-reference deep curve estimation for low-light image enhancement. In CVPR, 2020.
  10. Single image super-resolution from transformed self-exemplars. In CVPR, 2015.
  11. Enlightengan: Deep light enhancement without paired supervision. TIP, 2021.
  12. Darkvisionnet: Low-light imaging via rgb-nir fusion with deep inconsistency prior. In AAAI, 2022.
  13. Edge-based defocus blur estimation with adaptive scale selection. TIP, 2017.
  14. Image reconstruction with predictive filter flow. arXiv preprint arXiv:1811.11482, 2018.
  15. Iterative filter adaptive network for single image defocus deblurring. In CVPR, 2021.
  16. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022a.
  17. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint, 2023a.
  18. Close the loop: a unified bottom-up and top-down paradigm for joint image deraining and segmentation. In AAAI, 2022b.
  19. Efficient and explicit modelling of image hierarchies for image restoration. In CVPR, 2023b.
  20. When image denoising meets high-level vision tasks: A deep learning approach. In IJCAI, 2018.
  21. Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement. In CVPR, 2021.
  22. Toward fast, flexible, and robust low-light image enhancement. In CVPR, 2022.
  23. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, 2001.
  24. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR, 2017.
  25. Styleclip: Text-driven manipulation of stylegan imagery. In ICCV, 2021.
  26. Spatially-adaptive image restoration using distortion-guided networks. In ICCV, 2021.
  27. Learning transferable visual models from natural language supervision. In ICML, 2021.
  28. Real-world blur dataset for learning and benchmarking deblurring algorithms. In ECCV, 2020.
  29. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
  30. Human-aware motion deblurring. In ICCV, 2019.
  31. Just noticeable defocus blur detection and estimation. In CVPR, 2015.
  32. Purifying low-light images via near-infrared enlightened image. TMM, 2022.
  33. Exploring clip for assessing the look and feel of images. In AAAI, 2023a.
  34. Seeing dynamic scene in the dark: A high-quality video dataset with mechatronic alignment. In ICCV, 2021.
  35. Ultra-high-definition low-light image enhancement: A benchmark and transformer-based method. In AAAI, 2023b.
  36. Recovering realistic texture in image super-resolution by deep spatial feature transform. In CVPR, 2018.
  37. Uformer: A general u-shaped transformer for image restoration. In CVPR, 2022a.
  38. Blind2unblind: Self-supervised image denoising with visible blind spots. In CVPR, 2022b.
  39. Cris: Clip-driven referring image segmentation. In CVPR, 2022c.
  40. Uretinex-net: Retinex-based deep unfolding network for low-light image enhancement. In CVPR, 2022.
  41. Learning semantic-aware knowledge guidance for low-light image enhancement. In CVPR, 2023.
  42. Snr-aware low-light image enhancement. In CVPR, 2022a.
  43. Pvdd: A practical video denoising dataset with real-world dynamic scenes. arXiv preprint, 2022b.
  44. General adversarial defense against black-box attacks via pixel level and feature level distribution alignments. arXiv preprint, 2022c.
  45. Deep parametric 3d filters for joint video denoising and illumination enhancement in video super resolution. In AAAI, 2023a.
  46. Low-light image enhancement via structure modeling and guidance. In CVPR, 2023b.
  47. Ulip: Learning unified representation of language, image and point cloud for 3d understanding. In CVPR, 2023.
  48. Deep joint rain detection and removal from a single image. In CVPR, 2017.
  49. Sparse gradient regularized deep Retinex network for robust low-light image enhancement. TIP, 2021.
  50. K3dn: Disparity-aware kernel estimation for dual-pixel defocus deblurring. In CVPR, 2023.
  51. Multi-stage progressive image restoration. In CVPR, 2021.
  52. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, 2022.
  53. Lit: Zero-shot transfer with locked-image text tuning. In CVPR, 2022.
  54. Density-aware single image de-raining using a multi-stream dense network. In CVPR, 2018.
  55. Image de-raining using a conditional generative adversarial network. TCSVT, 2019.
  56. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. TIP, 2017.
  57. Plug-and-play image restoration with deep denoiser prior. TPAMI, 2021.
  58. Color demosaicking by local directional interpolation and nonlocal adaptive thresholding. JEI, 2011.
  59. Pointclip: Point cloud understanding by clip. In CVPR, 2022.
  60. Zegclip: Towards adapting clip for zero-shot semantic segmentation. In CVPR, 2023.
Citations (1)

Summary

We haven't generated a summary for this paper yet.