Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

SwinFuSR: an image fusion-inspired model for RGB-guided thermal image super-resolution (2404.14533v1)

Published 22 Apr 2024 in eess.IV and cs.CV

Abstract: Thermal imaging plays a crucial role in various applications, but the inherent low resolution of commonly available infrared (IR) cameras limits its effectiveness. Conventional super-resolution (SR) methods often struggle with thermal images due to their lack of high-frequency details. Guided SR leverages information from a high-resolution image, typically in the visible spectrum, to enhance the reconstruction of a high-res IR image from the low-res input. Inspired by SwinFusion, we propose SwinFuSR, a guided SR architecture based on Swin transformers. In real world scenarios, however, the guiding modality (e.g. RBG image) may be missing, so we propose a training method that improves the robustness of the model in this case. Our method has few parameters and outperforms state of the art models in terms of Peak Signal to Noise Ratio (PSNR) and Structural SIMilarity (SSIM). In Track 2 of the PBVS 2024 Thermal Image Super-Resolution Challenge, it achieves 3rd place in the PSNR metric. Our code and pretained weights are available at https://github.com/VisionICLab/SwinFuSR.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Acquisition of very high resolution images using stereo cameras. In Visual Communications and Image Processing’91: Visual Communication, pages 318–328. SPIE, 1991.
  2. A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets. The Visual Computer, 38(8):2939–2970, 2022.
  3. Multimodality video acquisition system for the assessment of vital distress in children. Sensors, 23(11):5293, 2023.
  4. Hemodynamic assessment in children after cardiac surgery: A pilot study on the value of infrared thermography. Frontiers in Pediatrics, 11, 2023.
  5. Monitoring of sugar beet growth indicators using wide-dynamic-range vegetation index (wdrvi) derived from uav multispectral images. Computers and Electronics in Agriculture, 171:105331, 2020.
  6. Simple Baselines for Image Restoration, 2022. arXiv:2204.04676 [cs].
  7. Activating more pixels in image super-resolution transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22367–22377, 2023.
  8. Infrared image super-resolution via locality-constrained group sparse model. Acta Physica Sinica, 63(4):044202–044202, 2014.
  9. Image super-resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence, 38(2):295–307, 2015.
  10. Accelerating the super-resolution convolutional neural network. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 391–407. Springer, 2016.
  11. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  12. Infrared image super-resolution via progressive compact distillation network. Electronics, 10(24):3107, 2021.
  13. Camera Array for Multi-Spectral Imaging. IEEE Transactions on Image Processing, 29:9234–9249, 2020. Conference Name: IEEE Transactions on Image Processing.
  14. MedSRGAN: medical images super-resolution using generative adversarial networks. Multimedia Tools and Applications, 79(29-30):21815–21840, 2020.
  15. First Science Results from SOFIA/FORCAST: Super-resolution Imaging of the S140 Cluster at 37 μ𝜇\muitalic_μm. The Astrophysical Journal Letters, 749(2):L20, 2012.
  16. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  17. Infrared Image Super-Resolution via Transfer Learning and PSRGAN. IEEE Signal Processing Letters, 28:982–986, 2021. Conference Name: IEEE Signal Processing Letters.
  18. Infrared Image Super-Resolution: Systematic Review, and Future Trends, 2022. arXiv:2212.12322 [cs, eess].
  19. Efficient and accurate quantized image super-resolution on mobile npus, mobile ai & aim 2022 challenge: Report. In Computer Vision – ECCV 2022 Workshops, pages 92–129, Cham, 2023. Springer Nature Switzerland.
  20. CoReFusion: Contrastive Regularized Fusion for Guided Thermal Super-Resolution. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 507–514, Vancouver, BC, Canada, 2023. IEEE.
  21. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1646–1654, 2016.
  22. Elastix: A Toolbox for Intensity-Based Medical Image Registration. IEEE Transactions on Medical Imaging, 29(1):196–205, 2010.
  23. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4681–4690, 2017.
  24. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1833–1844, 2021.
  25. DASR: Dual-Attention Transformer for infrared image super-resolution. Infrared Physics & Technology, 133:104837, 2023.
  26. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 136–144, 2017.
  27. Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5792–5801, New Orleans, LA, USA, 2022. IEEE.
  28. In-bed pose estimation: Deep learning with shallow dataset. IEEE Journal of Translational Engineering in Health and Medicine, 7:1–12, 2019.
  29. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021.
  30. Swinfusion: Cross-domain long-range learning for general image fusion via swin transformer. IEEE/CAA Journal of Automatica Sinica, 9(7):1200–1217, 2022.
  31. Smil: Multimodal learning with severely missing modality. ArXiv, abs/2103.05677, 2021.
  32. Image restoration using convolutional auto-encoders with symmetric skip connections. arXiv preprint arXiv:1606.08921, 2016.
  33. A super-resolution-based license plate recognition method for remote surveillance. Journal of Visual Communication and Image Representation, 94:103844, 2023.
  34. DA-VSR: Domain Adaptable Volumetric Super-Resolution For Medical Images, 2022. arXiv:2210.05117 [cs, eess].
  35. Estimation of peanut leaf area index from unmanned aerial vehicle multispectral images. Sensors, 20(23), 2020.
  36. Medical image super-resolution reconstruction algorithms based on deep learning: A survey. Computer Methods and Programs in Biomedicine, 238:107590, 2023.
  37. Discrete cosine transform based regularized high-resolution image reconstruction algorithm. Optical Engineering, 38(8):1348–1356, 1999.
  38. Thermal Image Super-Resolution Challenge Results - PBVS 2023. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 417–425, New Orleans, LA, USA, 2022. IEEE.
  39. U-net: Convolutional networks for biomedical image segmentation, 2015.
  40. A new approach for super-resolution and classification applications on neonatal thermal images. Quantitative InfraRed Thermography Journal, pages 1–18, 2023.
  41. Optical thermography infrastructure to assess thermal distribution in critically ill children. IEEE Open Journal of Engineering in Medicine and Biology, PP:1–1, 2021.
  42. A map approach for joint motion estimation, segmentation, and super resolution. IEEE Transactions on Image processing, 16(2):479–490, 2007.
  43. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1874–1883, 2016.
  44. Enhancement of guided thermal image super-resolution approaches. Neurocomputing, 573:127197, 2024.
  45. Ntire 2017 challenge on single image super-resolution: Methods and results. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1110–1121, 2017.
  46. Versatile near-infrared super-resolution imaging of amyloid fibrils with the fluorogenic probe cranad-2. Chemistry - A European Journal, 28, 2022.
  47. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  48. An analysis of a robust super resolution algorithm for infrared imaging. In 2009 Proceedings of 6th International Symposium on Image and Signal Processing and Analysis, pages 158–163. IEEE, 2009.
  49. Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis. In 2012 IEEE Conference on computer vision and pattern recognition, pages 2216–2223. IEEE, 2012.
  50. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European conference on computer vision (ECCV) workshops, pages 0–0, 2018.
  51. Towards good practices for missing modality robust action recognition, 2023.
  52. Data representation structure to support clinical decision-making in the pediatric intensive care unit: Interview study and preliminary decision support interface design. JMIR Formative Research, 8, 2024.
  53. Super-resolution imaging of the protoplanetary disk hd 142527 using sparse modeling. The Astrophysical Journal, 895(2):84, 2020.
  54. Coupled dictionary training for image super-resolution. IEEE transactions on image processing, 21(8):3467–3478, 2012.
  55. Deep networks with detail enhancement for infrared image super-resolution. IEEE Access, 8:158690–158701, 2020.
  56. SwinFIR: Revisiting the SwinIR with Fast Fourier Convolution and Improved Training for Image Super-Resolution, 2023a. arXiv:2208.11247 [cs].
  57. Efficient sparse representation based image super resolution via dual dictionary learning. In 2011 IEEE International Conference on Multimedia and Expo, pages 1–6. IEEE, 2011.
  58. Thermal Image Super-Resolution Based on Lightweight Dynamic Attention Network for Infrared Sensors. Sensors, 23(21):8717, 2023b.
  59. A review of deep learning for single image super-resolution. In 2019 International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS), pages 139–142, 2019.
  60. An infrared image super-resolution imaging algorithm based on auxiliary convolution neural network. In Other Conferences, 2020.
  61. Super-resolution reconstruction of infrared images based on a convolutional neural network with skip connections. Optics and Lasers in Engineering, 146:106717, 2021.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now