Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
GPT-4o
Gemini 2.5 Pro Pro
o3 Pro
GPT-4.1 Pro
DeepSeek R1 via Azure Pro
2000 character limit reached

Restore Anything Pipeline: Segment Anything Meets Image Restoration (2305.13093v2)

Published 22 May 2023 in cs.CV, cs.AI, cs.LG, and eess.IV

Abstract: Recent image restoration methods have produced significant advancements using deep learning. However, existing methods tend to treat the whole image as a single entity, failing to account for the distinct objects in the image that exhibit individual texture properties. Existing methods also typically generate a single result, which may not suit the preferences of different users. In this paper, we introduce the Restore Anything Pipeline (RAP), a novel interactive and per-object level image restoration approach that incorporates a controllable model to generate different results that users may choose from. RAP incorporates image segmentation through the recent Segment Anything Model (SAM) into a controllable image restoration model to create a user-friendly pipeline for several image restoration tasks. We demonstrate the versatility of RAP by applying it to three common image restoration tasks: image deblurring, image denoising, and JPEG artifact removal. Our experiments show that RAP produces superior visual results compared to state-of-the-art methods. RAP represents a promising direction for image restoration, providing users with greater control, and enabling image restoration at an object level.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. Ntire 2017 challenge on single image super-resolution: Dataset and study. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 126–135, 2017.
  2. Toward interactive modulation for photo-realistic image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 294–303, 2021.
  3. Progressive semantic-aware style transformation for blind face restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11896–11905, 2021.
  4. Simple baselines for image restoration. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VII, pages 17–33. Springer, 2022.
  5. Unsupervised lesion detection via image restoration with a normative prior. Medical image analysis, 64:101713, 2020.
  6. Segment and track anything. arXiv preprint arXiv:2305.06558, 2023.
  7. Is image super-resolution helpful for other vision tasks? In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1–9. IEEE, 2016.
  8. Remote sensing image super-resolution using novel dense-sampling networks. IEEE Transactions on Geoscience and Remote Sensing, 59(2):1618–1633, 2020.
  9. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
  10. Quantization guided jpeg artifact correction. In European Conference on Computer Vision, 2020.
  11. Jpeg artifacts reduction via deep convolutional sparse coding. In International Conference on Computer Vision, pages 2501–2510, 2019.
  12. Toward convolutional blind denoising of real photographs. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1712–1722, 2019.
  13. Task-driven super resolution: Object detection in low-resolution images. In Neural Information Processing: 28th International Conference, ICONIP 2021, Sanur, Bali, Indonesia, December 8–12, 2021, Proceedings, Part V 28, pages 387–395. Springer, 2021.
  14. Interactive multi-dimension modulation with dynamic controllable residual learning for image restoration. In European Conference on Computer Vision. Springer, 2020.
  15. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022.
  16. Towards flexible blind jpeg artifacts removal. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4997–5006, 2021.
  17. Edge-enhanced gan for remote sensing image superresolution. IEEE Transactions on Geoscience and Remote Sensing, 57(8):5799–5812, 2019.
  18. Imagic: Text-based real image editing with diffusion models. In Conference on Computer Vision and Pattern Recognition 2023, 2023.
  19. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
  20. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
  21. Deblurgan: Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8183–8192, 2018.
  22. Maskgan: Towards diverse and interactive facial image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5549–5558, 2020.
  23. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1833–1844, 2021.
  24. Editgan: High-precision semantic image editing. Advances in Neural Information Processing Systems, 34:16331–16345, 2021.
  25. Segment anything in medical images. arXiv preprint arXiv:2304.12306, 2023.
  26. Super-resolution capacitive touchscreens. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–10, 2021.
  27. Drag your gan: Interactive point-based manipulation on the generative image manifold. In ACM SIGGRAPH 2023 Conference Proceedings, 2023.
  28. Effects of image degradation and degradation removal to cnn-based image classification. IEEE transactions on pattern analysis and machine intelligence, 43(4):1239–1253, 2019.
  29. Neural blind deconvolution using deep priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3341–3350, 2020.
  30. Motion-from-blur: 3d shape and motion estimation of motion-blurred objects in videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15990–15999, 2022.
  31. Super-resolution microscopy with dna-paint. Nature protocols, 12(6):1198–1228, 2017.
  32. Towards privacy-preserving ego-motion estimation using an extremely low-resolution camera. IEEE Robotics and Automation Letters, 5(2):1223–1230, 2020.
  33. Capcontact: Super-resolution contact areas from capacitive touchscreens. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–14, 2021.
  34. Rethinking image restoration for object detection. Advances in Neural Information Processing Systems, 35:4461–4474, 2022.
  35. Ntire 2017 challenge on single image super-resolution: Methods and results. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 114–125, 2017.
  36. Dual super-resolution learning for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3774–3783, 2020.
  37. Cfsnet: Toward a controllable feature space for image restoration. In International Conference on Computer Vision, pages 4140–4149, 2019.
  38. Towards real-world blind face restoration with generative facial prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9168–9178, 2021.
  39. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1905–1914, 2021.
  40. Recovering realistic texture in image super-resolution by deep spatial feature transform. In IEEE Conference on Computer Vision and Pattern Recognition, pages 606–615, 2018.
  41. Deep degradation prior for low-quality image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11049–11058, 2020.
  42. Lg-bpn: Local and global blind-patch network for self-supervised real-world denoising. arXiv preprint arXiv:2304.00534, 2023.
  43. Edit everything: A text-guided generative system for images editing. arXiv preprint arXiv:2304.14006, 2023.
  44. Inpaint anything: Segment anything meets image inpainting. arXiv preprint arXiv:2304.06790, 2023.
  45. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5728–5739, 2022.
  46. Plug-and-play image restoration with deep denoiser prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6360–6376, 2021.
  47. Designing a practical degradation model for deep blind image super-resolution. In IEEE Conference on International Conference on Computer Vision, 2021.
  48. Deblurring by realistic blurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2737–2746, 2020.
  49. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing, 26(7):3142–3155, 2017.
  50. Ffdnet: Toward a fast and flexible solution for cnn-based image denoising. IEEE Transactions on Image Processing, 27(9):4608–4622, 2018.
  51. Learning a single convolutional super-resolution network for multiple degradations. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3262–3271, 2018.
  52. Residual dense network for image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2472–2481, 2018.
Citations (7)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now