Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

TransMRSR: Transformer-based Self-Distilled Generative Prior for Brain MRI Super-Resolution (2306.06669v1)

Published 11 Jun 2023 in eess.IV, cs.CV, and cs.LG

Abstract: Magnetic resonance images (MRI) acquired with low through-plane resolution compromise time and cost. The poor resolution in one orientation is insufficient to meet the requirement of high resolution for early diagnosis of brain disease and morphometric study. The common Single image super-resolution (SISR) solutions face two main challenges: (1) local detailed and global anatomical structural information combination; and (2) large-scale restoration when applied for reconstructing thick-slice MRI into high-resolution (HR) iso-tropic data. To address these problems, we propose a novel two-stage network for brain MRI SR named TransMRSR based on the convolutional blocks to extract local information and transformer blocks to capture long-range dependencies. TransMRSR consists of three modules: the shallow local feature extraction, the deep non-local feature capture, and the HR image reconstruction. We perform a generative task to encapsulate diverse priors into a generative network (GAN), which is the decoder sub-module of the deep non-local feature capture part, in the first stage. The pre-trained GAN is used for the second stage of SR task. We further eliminate the potential latent space shift caused by the two-stage training strategy through the self-distilled truncation trick. The extensive experiments show that our method achieves superior performance to other SSIR methods on both public and private datasets. Code is released at https://github.com/goddesshs/TransMRSR.git .

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Y. Xia, N. Ravikumar, J. P. Greenwood, S. Neubauer, S. E. Petersen, and A. F. Frangi, “Super-resolution of cardiac mr cine imaging using conditional gans and unsupervised transfer learning,” Medical Image Analysis, vol. 71, p. 102037, 2021.
  2. C. Zhao, B. E. Dewey, D. L. Pham, P. A. Calabresi, D. S. Reich, and J. L. Prince, “Smore: a self-supervised anti-aliasing and super-resolution algorithm for mri using deep learning,” IEEE transactions on medical imaging, vol. 40, no. 3, pp. 805–817, 2020.
  3. G. Liu, Z. Cao, Q. Xu, Q. Zhang, F. Yang, X. Xie, J. Hao, Y. Shi, B. C. Bernhardt, Y. He et al., “Recycling diagnostic mri for empowering brain morphometric research–critical & practical assessment on learning-based image super-resolution,” Neuroimage, vol. 245, p. 118687, 2021.
  4. C. Peng, W.-A. Lin, H. Liao, R. Chellappa, and S. K. Zhou, “Saint: spatially aware interpolation network for medical slice synthesis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 7750–7759.
  5. Y. Zhang, K. Li, K. Li, and Y. Fu, “Mr image super-resolution with squeeze and excitation reasoning attention network,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 13 425–13 434.
  6. Y. Chen, Y. Xie, Z. Zhou, F. Shi, A. G. Christodoulou, and D. Li, “Brain mri super resolution using 3d deep densely connected neural networks,” in 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018).   IEEE, 2018, pp. 739–742.
  7. J. Du, L. Wang, Y. Liu, Z. Zhou, Z. He, and Y. Jia, “Brain mri super-resolution using 3d dilated convolutional encoder–decoder network,” IEEE Access, vol. 8, pp. 18 938–18 950, 2020.
  8. J. Wang, Y. Chen, Y. Wu, J. Shi, and J. Gee, “Enhanced generative adversarial network for 3d brain mri super-resolution,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2020, pp. 3627–3636.
  9. Y. Chen, F. Shi, A. G. Christodoulou, Y. Xie, Z. Zhou, and D. Li, “Efficient and accurate mri super-resolution using a generative adversarial network and 3d multi-level densely connected network,” in Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part I.   Springer, 2018, pp. 91–99.
  10. G. Li, J. Lv, Y. Tian, Q. Dou, C. Wang, C. Xu, and J. Qin, “Transformer-empowered multi-scale contextual matching and aggregation for multi-contrast mri super-resolution,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 20 636–20 645.
  11. B. Zhang, S. Gu, B. Zhang, J. Bao, D. Chen, F. Wen, Y. Wang, and B. Guo, “Styleswin: Transformer-based gan for high-resolution image generation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 11 304–11 314.
  12. C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE transactions on pattern analysis and machine intelligence, vol. 38, no. 2, pp. 295–307, 2015.
  13. X. Liu, L. Chen, W. Wang, and J. Zhao, “Robust multi-frame super-resolution based on spatially weighted half-quadratic estimation and adaptive btv regularization,” IEEE Transactions on Image Processing, vol. 27, no. 10, pp. 4971–4986, 2018.
  14. X. Liu, L. Kong, Y. Zhou, J. Zhao, and J. Chen, “End-to-end trainable video super-resolution based on a new mechanism for implicit motion estimation and compensation,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2020, pp. 2416–2425.
  15. W. Wang, J. Hu, X. Liu, J. Zhao, and J. Chen, “Single image super resolution based on multi-scale structure and non-local smoothing,” EURASIP Journal on Image and Video Processing, vol. 2021, no. 1, p. 16, 2021.
  16. Z. Shi, X. Liu, C. Li, L. Dai, J. Chen, T. N. Davidson, and J. Zhao, “Learning for unconstrained space-time video super-resolution,” IEEE Transactions on Broadcasting, vol. 68, no. 2, pp. 345–358, 2021.
  17. X. Liu, K. Shi, Z. Wang, and J. Chen, “Exploit camera raw data for video super-resolution via hidden markov model inference,” IEEE Transactions on Image Processing, vol. 30, pp. 2127–2140, 2021.
  18. X. Chu, L. Chen, and W. Yu, “Nafssr: stereo image super-resolution using nafnet,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 1239–1248.
  19. J. Kim, J. K. Lee, and K. M. Lee, “Accurate image super-resolution using very deep convolutional networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1646–1654.
  20. W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1874–1883.
  21. C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang et al., “Photo-realistic single image super-resolution using a generative adversarial network,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4681–4690.
  22. C.-M. Feng, H. Fu, S. Yuan, and Y. Xu, “Multi-contrast mri super-resolution via a multi-stage integration network,” in Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VI 24.   Springer, 2021, pp. 140–149.
  23. Q. Lyu, H. Shan, and G. Wang, “Mri super-resolution with ensemble learning and complementary priors,” IEEE Transactions on Computational Imaging, vol. 6, pp. 615–624, 2020.
  24. H. Zhang, H. Li, D. Zhang, Y. Zhang, X. Wang, Y. Xia, Y. Shi, and W. Wang, “Mri super-resolution using 3d deeply residual and densely convolutional neural networks,” IEEE Transactions on Medical Imaging, vol. 38, no. 1, pp. 167–179, 2019.
  25. S. Chartier, A. M. Khairy, M. Reisert, S. Meriaux, J. Montagnat, and H. Liebgott, “Multi-scale 3d generative adversarial networks for mr image synthesis,” in International Conference on Medical Image Computing and Computer-Assisted Intervention.   Springer, 2018, pp. 198–206.
  26. Z. Jin, Y. Li, W. Chen, H. Liu, Y. Zhang, and Q. Zhang, “Deep learning-based 3d mri super-resolution with multiple inference paths,” IEEE Journal of Biomedical and Health Informatics, 2022.
  27. D. Bau, H. Strobelt, W. Peebles, J. Wulff, B. Zhou, J.-Y. Zhu, and A. Torralba, “Semantic photo manipulation with a generative image prior,” arXiv preprint arXiv:2005.07727, 2020.
  28. L. Goetschalckx, A. Andonian, A. Oliva, and P. Isola, “Ganalyze: Toward visual definitions of cognitive image properties,” in Proceedings of the ieee/cvf international conference on computer vision, 2019, pp. 5744–5753.
  29. W. Xia, Y. Zhang, Y. Yang, J.-H. Xue, B. Zhou, and M.-H. Yang, “Gan inversion: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  30. S. Menon, A. Damian, S. Hu, N. Ravi, and C. Rudin, “Pulse: Self-supervised photo upsampling via latent space exploration of generative models,” in Proceedings of the ieee/cvf conference on computer vision and pattern recognition, 2020, pp. 2437–2445.
  31. J. Gu, Y. Shen, and B. Zhou, “Image processing using multi-code gan prior,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 3012–3021.
  32. J. Zhu, Y. Shen, D. Zhao, and B. Zhou, “In-domain gan inversion for real image editing,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVII 16.   Springer, 2020, pp. 592–608.
  33. K. C. Chan, X. Wang, X. Xu, J. Gu, and C. C. Loy, “Glean: Generative latent bank for large-factor image super-resolution,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 14 245–14 254.
  34. X. Wang, Y. Li, H. Zhang, and Y. Shan, “Towards real-world blind face restoration with generative facial prior,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 9168–9178.
  35. Y. Korkmaz, S. U. Dar, M. Yurt, M. Özbey, and T. Cukur, “Unsupervised mri reconstruction via zero-shot learned adversarial transformers,” IEEE Transactions on Medical Imaging, vol. 41, no. 7, pp. 1747–1763, 2022.
  36. T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 4401–4410.
  37. R. Mokady, O. Tov, M. Yarom, O. Lang, I. Mosseri, T. Dekel, D. Cohen-Or, and M. Irani, “Self-distilled stylegan: Towards generation from internet photos,” in ACM SIGGRAPH 2022 Conference Proceedings, 2022, pp. 1–9.
  38. J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte, “Swinir: Image restoration using swin transformer,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 1833–1844.
  39. J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14.   Springer, 2016, pp. 694–711.
  40. L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2414–2423.
  41. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022.
  42. B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee, “Enhanced deep residual networks for single image super-resolution,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2017, pp. 136–144.
  43. S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang, “Restormer: Efficient transformer for high-resolution image restoration,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5728–5739.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Shan Huang (69 papers)
  2. Xiaohong Liu (117 papers)
  3. Tao Tan (54 papers)
  4. Menghan Hu (21 papers)
  5. Xiaoer Wei (1 paper)
  6. Tingli Chen (1 paper)
  7. Bin Sheng (38 papers)
Citations (30)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now