Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
143 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Selective Hourglass Mapping for Universal Image Restoration Based on Diffusion Model (2403.11157v1)

Published 17 Mar 2024 in cs.CV

Abstract: Universal image restoration is a practical and potential computer vision task for real-world applications. The main challenge of this task is handling the different degradation distributions at once. Existing methods mainly utilize task-specific conditions (e.g., prompt) to guide the model to learn different distributions separately, named multi-partite mapping. However, it is not suitable for universal model learning as it ignores the shared information between different tasks. In this work, we propose an advanced selective hourglass mapping strategy based on diffusion model, termed DiffUIR. Two novel considerations make our DiffUIR non-trivial. Firstly, we equip the model with strong condition guidance to obtain accurate generation direction of diffusion model (selective). More importantly, DiffUIR integrates a flexible shared distribution term (SDT) into the diffusion algorithm elegantly and naturally, which gradually maps different distributions into a shared one. In the reverse process, combined with SDT and strong condition guidance, DiffUIR iteratively guides the shared distribution to the task-specific distribution with high image quality (hourglass). Without bells and whistles, by only modifying the mapping strategy, we achieve state-of-the-art performance on five image restoration tasks, 22 benchmarks in the universal setting and zero-shot generalization setting. Surprisingly, by only using a lightweight model (only 0.89M), we could achieve outstanding performance. The source code and pre-trained models are available at https://github.com/iSEE-Laboratory/DiffUIR

Definition Search Book Streamline Icon: https://streamlinehq.com
References (75)
  1. Blended diffusion for text-driven editing of natural images. In CVPR, 2022.
  2. Label-efficient semantic segmentation with diffusion models. In ICLR, 2021.
  3. Depth camera based localization and navigation for indoor mobile robots. In RSS, 2011.
  4. Deepdriving: Learning affordance for direct perception in autonomous driving. In ICCV, 2015.
  5. Hinet: Half instance normalization network for image restoration. In CVPR, 2021.
  6. Simple baselines for image restoration. In ECCV, 2022.
  7. Diffusiondet: Diffusion model for object detection. In ICCV, 2023a.
  8. Learning a sparse transformer network for effective image deraining. In CVPR, 2023b.
  9. Diffedit: Diffusion-based semantic image editing with mask guidance. In ICLR, 2022.
  10. Inversion by direct iteration: An alternative to denoising diffusion for image restoration. arXiv preprint arXiv:2303.11435, 2023.
  11. A general decoupled learning framework for parameterized image operators. TPAMI, 2019.
  12. Single image haze removal using dark channel prior. TPAMI, 2010.
  13. Deep residual learning for image recognition. In CVPR, 2016.
  14. Denoising diffusion probabilistic models. In NeurIPS, 2020.
  15. Scope of validity of psnr in image/video quality assessment. EL, 2008.
  16. Multi-scale progressive fusion network for single image deraining. In CVPR, 2020a.
  17. Multi-scale progressive fusion network for single image deraining. In CVPR, 2020b.
  18. Adam: A method for stochastic optimization. In ICLR, 2015.
  19. Auto-encoding variational bayes. In ICLR, 2014.
  20. An introduction to variational autoencoders. FTML, 2019.
  21. Contrast enhancement based on layered difference representation. In ICIP, 2012.
  22. Benchmarking single-image dehazing and beyond. TIP, 2018.
  23. All-in-one image restoration for unknown corruption. In CVPR, 2022.
  24. All in one bad weather removal using architectural search. In CVPR, 2020.
  25. Swinir: Image restoration using swin transformer. In ICCV, 2021.
  26. Iterative prompt learning for unsupervised backlit image enhancement. In ICCV, 2023.
  27. I22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT sb: Image-to-image schrödinger bridge. arXiv preprint arXiv:2302.05872, 2023a.
  28. Residual denoising diffusion models. arXiv preprint arXiv:2308.13712, 2023b.
  29. Tape: Task-agnostic prior embedding for image restoration. In ECCV, 2022.
  30. Desnownet: Context-aware deep network for snow removal. TIP, 2018.
  31. Controlling vision-language models for universal image restoration. arXiv preprint arXiv:2310.01018, 2023a.
  32. Image restoration with mean-reverting stochastic differential equations. ICML, 2023b.
  33. Prores: Exploring degradation-aware visual prompt for universal image restoration. arXiv preprint arXiv:2306.13653, 2023.
  34. Perceptual quality assessment for multi-exposure image fusion. TIP, 2015.
  35. Making a “completely blind” image quality analyzer. SPL, 2012.
  36. Deep generalized unfolding networks for image restoration. In CVPR, 2022.
  37. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR, 2017.
  38. Restoring vision in adverse weather conditions with patch-based denoising diffusion models. TPAMI, 2023.
  39. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
  40. Promptir: Prompting for all-in-one blind image restoration. In NeurIPS, 2023.
  41. Spatially-adaptive image restoration using distortion-guided networks. In ICCV, 2021.
  42. Real-world blur dataset for learning and benchmarking deblurring algorithms. In ECCV, 2020.
  43. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
  44. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
  45. Palette: Image-to-image diffusion models. In SIGGRAPH, 2022.
  46. Diffustereo: High quality human reconstruction via diffusion-based stereo using sparse cameras. In ECCV, 2022.
  47. Human-aware motion deblurring. In ICCV, 2019.
  48. Denoising diffusion implicit models. In ICLR, 2020a.
  49. Generative modeling by estimating gradients of the data distribution. In NeurIPS, 2019.
  50. Score-based generative modeling through stochastic differential equations. In ICLR, 2020b.
  51. Vision transformers for single image dehazing. TIP, 2023.
  52. Maxim: Multi-axis mlp for image processing. In CVPR, 2022.
  53. Transweather: Transformer-based restoration of images degraded by adverse weather conditions. In CVPR, 2022.
  54. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008.
  55. Pdpp: Projected diffusion for procedure planning in instructional videos. In CVPR, 2023a.
  56. Naturalness preserved enhancement algorithm for non-uniform illumination images. TIP, 2013.
  57. Images speak in images: A generalist painter for in-context visual learning. In CVPR, 2023b.
  58. Zero-shot image restoration using denoising diffusion null-space model. In ICLR, 2022a.
  59. Image quality assessment: from error visibility to structural similarity. TIP, 2004.
  60. Restoreformer: High-quality blind face restoration from undegraded key-value pairs. In CVPR, 2022b.
  61. Deep retinex decomposition for low-light enhancement. In BMVC, 2018.
  62. Raindiffusion: When unsupervised learning meets diffusion models for real-world image deraining. arXiv preprint arXiv:2301.09430, 2023.
  63. Finding discriminative filters for specific degradations in blind super-resolution. In NeurIPS, 2021.
  64. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR, 2023.
  65. Implicit neural representation for cooperative low-light image enhancement. In ICCV, 2023.
  66. Deep joint rain detection and removal from a single image. In CVPR, 2017.
  67. Freedom: Training-free energy-guided conditional diffusion model. ICCV, 2023.
  68. Multi-stage progressive image restoration. In CVPR, 2021a.
  69. Multi-stage progressive image restoration. In CVPR, 2021b.
  70. Learning enriched features for fast image restoration and enhancement. TPAMI, 2022.
  71. Ingredient-oriented multi-degradation learning for image restoration. In CVPR, 2023.
  72. A feature-enriched completely blind image quality evaluator. TIP, 2015.
  73. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
  74. Diffuvolume: Diffusion model for volume based stereo matching. arXiv preprint arXiv:2308.15989, 2023.
  75. Image restoration for under-display camera. In CVPR, 2021.
Citations (13)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now