Papers
Topics
Authors
Recent
Search
2000 character limit reached

Deep learning techniques for blind image super-resolution: A high-scale multi-domain perspective evaluation

Published 15 Jun 2023 in eess.IV, cs.CV, and cs.LG | (2306.09426v1)

Abstract: Despite several solutions and experiments have been conducted recently addressing image super-resolution (SR), boosted by deep learning (DL) techniques, they do not usually design evaluations with high scaling factors, capping it at 2x or 4x. Moreover, the datasets are generally benchmarks which do not truly encompass significant diversity of domains to proper evaluate the techniques. It is also interesting to remark that blind SR is attractive for real-world scenarios since it is based on the idea that the degradation process is unknown, and hence techniques in this context rely basically on low-resolution (LR) images. In this article, we present a high-scale (8x) controlled experiment which evaluates five recent DL techniques tailored for blind image SR: Adaptive Pseudo Augmentation (APA), Blind Image SR with Spatially Variant Degradations (BlindSR), Deep Alternating Network (DAN), FastGAN, and Mixture of Experts Super-Resolution (MoESR). We consider 14 small datasets from five different broader domains which are: aerial, fauna, flora, medical, and satellite. Another distinctive characteristic of our evaluation is that some of the DL approaches were designed for single-image SR but others not. Two no-reference metrics were selected, being the classical natural image quality evaluator (NIQE) and the recent transformer-based multi-dimension attention network for no-reference image quality assessment (MANIQA) score, to assess the techniques. Overall, MoESR can be regarded as the best solution although the perceptual quality of the created HR images of all the techniques still needs to improve. Supporting code: https://github.com/vsantjr/DL_BlindSR. Datasets: https://www.kaggle.com/datasets/valdivinosantiago/dl-blindsr-datasets.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. K. Yamashita and K. Markov. Medical image enhancement using super resolution methods. In Valeria V. Krzhizhanovskaya, Gábor Závodszky, Michael H. Lees, Jack J. Dongarra, Peter M. A. Sloot, Sérgio Brissos, and João Teixeira, editors, Computational Science – ICCS 2020, pages 496–508, Cham, 2020. Springer International Publishing.
  2. Simultaneous super-resolution and cross-modality synthesis of 3d medical images using weakly-supervised joint convolutional sparse coding. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5787–5796, Los Alamitos, CA, USA, jul 2017. IEEE Computer Society.
  3. A new generative adversarial network for medical images super resolution. Scientific Reports, 12:9533, 2022. https://doi.org/10.1038/s41598-022-13658-4.
  4. Super-resolution using gans for medical imaging. Procedia Computer Science, 173:28–35, 2020. International Conference on Smart Sustainable Intelligent Computing and Applications under ICITETM2020.
  5. Internet video delivery improved by super-resolution with gan. Future Internet, 14(12), 2022.
  6. Neural adaptive content-aware internet video delivery. In Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementation, OSDI’18, pages 645–661, USA, 2018. USENIX Association.
  7. How will deep learning change internet video delivery? In HotNets-XVI: Proceedings of the 16th ACM Workshop on Hot Topics in Networks, pages 57–64, 2017.
  8. Convolutional neural network super resolution for face recognition in surveillance monitoring. In Francisco José Perales and Josef Kittler, editors, Articulated Motion and Deformable Objects, pages 175–184, Cham, 2016. Springer International Publishing.
  9. Learning face hallucination in the wild. Proceedings of the AAAI Conference on Artificial Intelligence, 29(1), Mar. 2015.
  10. Deep cascaded bi-network for face hallucination. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 614–630, Cham, 2016. Springer International Publishing.
  11. A comprehensive review on deep learning based remote sensing image super-resolution methods. Earth-Science Reviews, 232:104110, 2022.
  12. Edge-enhanced gan for remote sensing image superresolution. IEEE Transactions on Geoscience and Remote Sensing, 57(8):5799–5812, 2019.
  13. Deep distillation recursive network for remote sensing imagery super-resolution. Remote Sensing, 10(11), 2018.
  14. Te-sagan: An improved generative adversarial network for remote sensing super-resolution images. Remote Sensing, 14(10), 2022.
  15. Single image super-resolution via a holistic attention network. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, pages 191–207, Cham, 2020. Springer International Publishing.
  16. Single-image super resolution of remote sensing images with real-world degradation modeling. Remote Sensing, 14(12), 2022.
  17. Deep learning for image super-resolution: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(10):3365–3387, 2021.
  18. C. E. Duchon. Lanczos filtering in one and two dimensions. Journal of Applied Meteorology and Climatology, 18(8):1016 – 1022, 1979.
  19. Image super-resolution using gradient profile prior. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8, 2008.
  20. Robust web image/video super-resolution. IEEE Transactions on Image Processing, 19(8):2017–2028, 2010.
  21. Deep learning-a first meta-survey of selected reviews across scientific disciplines, their commonalities, challenges and research impact. PeerJ Computer science, 7, 2021. © 2021 Egger et al.
  22. Learning a deep convolutional network for image super-resolution. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, pages 184–199, Cham, 2014. Springer International Publishing.
  23. Accurate image super-resolution using very deep convolutional networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1646–1654, Los Alamitos, CA, USA, jun 2016. IEEE Computer Society.
  24. Enhanced deep residual networks for single image super-resolution. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1132–1140, 2017.
  25. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  26. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, September 2018.
  27. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  28. Sesr: Single image super resolution with recursive squeeze and excitation networks. In 2018 24th International Conference on Pattern Recognition (ICPR), pages 147–152, 2018.
  29. To learn image super-resolution, use a gan to learn how to do image degradation first. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision – ECCV 2018, pages 187–202, Cham, 2018. Springer International Publishing.
  30. Blind super-resolution with iterative kernel correction. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1604–1613, Los Alamitos, CA, USA, jun 2019. IEEE Computer Society.
  31. Blind image super-resolution with elaborate degradation modeling on noise and kernel. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2118–2128, Los Alamitos, CA, USA, jun 2022. IEEE Computer Society.
  32. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pages 416–423 vol.2, 2001.
  33. Contour detection and hierarchical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(5):898–916, 2011.
  34. E. Agustsson and R. Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1122–1131, 2017.
  35. The 2018 pirm challenge on perceptual image super-resolution. In Laura Leal-Taixé and Stefan Roth, editors, Computer Vision – ECCV 2018 Workshops, pages 334–355, Cham, 2019. Springer International Publishing.
  36. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In R. Bowden, J. Collomosse, and K. Mikolajczyk, editors, Proceedings of the British Machine Vision Conference, pages 1–10, Durham, UK, 2012. BMVA Press.
  37. On single image scale-up using sparse-representations. In Jean-Daniel Boissonnat, Patrick Chenin, Albert Cohen, Christian Gout, Tom Lyche, Marie-Laurence Mazure, and Larry Schumaker, editors, Curves and Surfaces, pages 711–730, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg.
  38. Single image super-resolution from transformed self-exemplars. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5197–5206, 2015.
  39. Few-shot remote sensing image scene classification based on metric learning and local descriptors. Remote Sensing, 15(3), 2023.
  40. Deep back-projection networks for super-resolution. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1664–1673, Los Alamitos, CA, USA, jun 2018. IEEE Computer Society.
  41. Ntire 2022 challenge on perceptual image quality assessment. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 950–966, Los Alamitos, CA, USA, jun 2022. IEEE Computer Society.
  42. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  43. Making a “completely blind” image quality analyzer. IEEE Signal Processing Letters, 20(3):209–212, 2013.
  44. Deceive d: Adaptive pseudo augmentation for gan training with limited data. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 21655–21667. Curran Associates, Inc., 2021.
  45. Blind image super-resolution with spatially variant degradations. ACM Trans. Graph., 38(6), nov 2019.
  46. Unfolding the alternating optimization for blind super resolution. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 5632–5643. Curran Associates, Inc., 2020.
  47. Towards faster and stabilized {gan} training for high-fidelity few-shot image synthesis. In International Conference on Learning Representations, 2021.
  48. Moesr: Blind super-resolution using kernel-aware mixture of experts. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 4009–4018, 2022.
  49. Maniqa: Multi-dimension attention network for no-reference image quality assessment, 2022.
  50. Learning a single convolutional super-resolution network for multiple degradations. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3262–3271, 2018.
  51. Joint blind source separation with multivariate gaussian model: Algorithms and performance analysis. IEEE Transactions on Signal Processing, 60(4):1672–1683, 2012.
  52. Image processing using multi-code gan prior. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3012—3021, June 2020.
  53. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
  54. Generalizing from a few examples: A survey on few-shot learning. ACM Comput. Surv., 53(3), jun 2020.
  55. M. Irani A. Shocher, N. Cohen. ”Zero-Shot” super-resolution using deep internal learning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  56. Blind Super-Resolution Kernel Estimation Using an Internal-GAN. Curran Associates Inc., Red Hook, NY, USA, 2019.
  57. Training generative adversarial networks with limited data. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc.
  58. Differentiable augmentation for data-efficient gan training. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc.
  59. Veegan: Reducing mode collapse in gans using implicit variational learning. Neural Information Processing Systems, 2017.
  60. A. Mackiewicz and W. Ratajczak. Principal components analysis (pca). Computers & Geosciences, 19(3):303–342, 1993.
  61. E. Elhaik. Principal component analyses (pca)-based findings in population genetic studies are highly biased and must be reevaluated. Scientific Reports, 12(14683), 2022. https://doi.org/10.1038/s41598-022-14395-4.

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up