Papers
Topics
Authors
Recent
Search
2000 character limit reached

LIT-Former: Linking In-plane and Through-plane Transformers for Simultaneous CT Image Denoising and Deblurring

Published 21 Feb 2023 in eess.IV, cs.CV, and physics.med-ph | (2302.10630v2)

Abstract: This paper studies 3D low-dose computed tomography (CT) imaging. Although various deep learning methods were developed in this context, typically they focus on 2D images and perform denoising due to low-dose and deblurring for super-resolution separately. Up to date, little work was done for simultaneous in-plane denoising and through-plane deblurring, which is important to obtain high-quality 3D CT images with lower radiation and faster imaging speed. For this task, a straightforward method is to directly train an end-to-end 3D network. However, it demands much more training data and expensive computational costs. Here, we propose to link in-plane and through-plane transformers for simultaneous in-plane denoising and through-plane deblurring, termed as LIT-Former, which can efficiently synergize in-plane and through-plane sub-tasks for 3D CT imaging and enjoy the advantages of both convolution and transformer networks. LIT-Former has two novel designs: efficient multi-head self-attention modules (eMSM) and efficient convolutional feedforward networks (eCFN). First, eMSM integrates in-plane 2D self-attention and through-plane 1D self-attention to efficiently capture global interactions of 3D self-attention, the core unit of transformer networks. Second, eCFN integrates 2D convolution and 1D convolution to extract local information of 3D convolution in the same fashion. As a result, the proposed LIT-Former synergize these two subtasks, significantly reducing the computational complexity as compared to 3D counterparts and enabling rapid convergence. Extensive experimental results on simulated and clinical datasets demonstrate superior performance over state-of-the-art models. The source code is made available at https://github.com/hao1635/LIT-Former.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. N. B. Shah and S. L. Platt, “ALARA: is there a cause for alarm? Reducing radiation risks from computed tomography scanning in children,” Current Opinion in Pediatrics, vol. 20, no. 3, pp. 243–247, 2008.
  2. S. Park, S. M. Lee, K.-H. Do, J.-G. Lee, W. Bae, H. Park, K.-H. Jung, and J. B. Seo, “Deep learning algorithm for reducing ct slice thickness: effect on reproducibility of radiomic features in lung cancer,” Korean J. Radiol., vol. 20, no. 10, pp. 1431–1440, 2019.
  3. F. Fischbach, F. Knollmann, V. Griesshaber, T. Freund, E. Akkol, and R. Felix, “Detection of pulmonary nodules by multislice computed tomography: improved detection rate with reduced slice thickness,” European Radio., vol. 13, pp. 2378–2383, 2003.
  4. G. Wang, “A perspective on deep imaging,” IEEE Access, vol. 4, pp. 8914–8924, 2016.
  5. G. Wang, J. C. Ye, K. Mueller, and J. A. Fessler, “Image reconstruction is a new frontier of machine learning,” IEEE Trans. Med. Imag., vol. 37, no. 6, pp. 1289–1296, 2018.
  6. G. Wang, J. C. Ye, and B. De Man, “Deep learning for tomographic image reconstruction,” Nat. Mach. Intell., vol. 2, no. 12, pp. 737–748, 2020.
  7. E. Kang, J. Min, and J. C. Ye, “A deep convolutional neural network using directional wavelets for low-dose X-ray CT reconstruction,” Med. Phys., vol. 44, no. 10, pp. e360–e375, 2017.
  8. H. Chen et al., “Low-dose CT via convolutional neural network,” Biomed. Opt. Express, vol. 8, no. 2, pp. 679–694, 2017.
  9. H. Chen et al., “Low-dose CT with a residual encoder-decoder convolutional neural network,” IEEE Trans. Med. Imaging, vol. 36, no. 12, pp. 2524–2535, 2017.
  10. H. Shan et al., “3-D convolutional encoder-decoder network for low-dose CT via transfer learning from a 2-D trained network,” IEEE Trans. Med. Imaging, vol. 37, no. 6, pp. 1522–1534, 2018.
  11. Q. Yang et al., “Low-dose CT image denoising using a generative adversarial network with wasserstein distance and perceptual loss,” IEEE Trans. Med. Imaging, vol. 37, no. 6, pp. 1348–1357, 2018.
  12. H. Shan et al., “Competitive performance of a modularized deep neural network compared to commercial algorithms for low-dose CT image reconstruction,” Nat. Mach. Intell., vol. 1, no. 6, pp. 269–276, 2019.
  13. C. Niu et al., “Noise suppression with similarity-based self-supervised deep learning,” IEEE Trans. Med. Imaging, 2022.
  14. T. Liang, Y. Jin, Y. Li, and T. Wang, “EDCNN: Edge enhancement-based densely connected network with compound loss for low-dose CT denoising,” in IEEE Int. Conf. Signal Process., vol. 1.   IEEE, 2020, pp. 193–198.
  15. Z. Huang, J. Zhang, Y. Zhang, and H. Shan, “DU-GAN: Generative adversarial networks with dual-domain U-Net-based discriminators for low-dose CT denoising,” IEEE Trans. Instrum. Meas., vol. 71, pp. 1–12, 2021.
  16. H. Liu, X. Jin, and L. Liu, “Low-dose CT image denoising based on improved DD-Net and local filtered mechanism,” Comput. Intell. Neurosci., vol. 2022, 2022.
  17. D. Wang, F. Fan, Z. Wu, R. Liu, F. Wang, and H. Yu, “CTformer: Convolution-free token2token dilated vision transformer for low-dose CT denoising,” Phys. Med. Biol., 2023.
  18. J. Park, D. Hwang, K. Y. Kim, S. K. Kang, Y. K. Kim, and J. S. Lee, “Computed tomography super-resolution using deep convolutional neural network,” Phys. Med. Biol., vol. 63, no. 14, p. 145011, 2018.
  19. C. You et al., “CT super-resolution GAN constrained by the identical, residual, and cycle learning ensemble (GAN-CIRCLE),” IEEE Trans. Med. Imaging, vol. 39, no. 1, pp. 188–203, 2019.
  20. X. Zhang, C. Feng, A. Wang, L. Yang, and Y. Hao, “CT super-resolution using multiple dense residual block based GAN,” Signal Image Video Process., vol. 15, no. 4, pp. 725–733, 2021.
  21. N. J. Pelc, “Recent and future directions in ct imaging,” Ann. Biomed. Eng ., vol. 42, pp. 260–268, 2014.
  22. H. Wang, L.-L. Li, J. Shang, J. Song, and B. Liu, “Application of deep learning image reconstruction in low-dose chest ct scan,” Br. J. Radiol., vol. 95, no. 1133, p. 20210380, 2022.
  23. Y. Xiao, A. Gupta, P. C. Sanelli, and R. Fang, “STAR: spatio-temporal architecture for super-resolution in low-dose CT perfusion,” in Mach. Learn. Med. Imag.   Springer, 2017, pp. 97–105.
  24. D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun, and M. Paluri, “A closer look at spatiotemporal convolutions for action recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 6450–6459.
  25. Z. Liu, L. Wang, W. Wu, C. Qian, and T. Lu, “TAM: Temporal adaptive module for video recognition,” in Proc. IEEE Int. Conf. Comput. Vis., 2021, pp. 13 708–13 718.
  26. Z. Huang et al., “TAda! temporally-adaptive convolutions for video understanding,” in Proc. Int. Conf. Learn. Represent., 2022.
  27. S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang, “Restormer: Efficient transformer for high-resolution image restoration,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2022, pp. 5728–5739.
  28. A. Vaswani et al., “Attention is all you need,” Adv. Neural Inf. Process. Syst., vol. 30, 2017.
  29. Z. Liu et al., “Swin Transformer: Hierarchical vision transformer using shifted windows,” in Proc. IEEE Int. Conf. Comput. Vis., 2021, pp. 10 012–10 022.
  30. Z. Wang, X. Cun, J. Bao, W. Zhou, J. Liu, and H. Li, “Uformer: A general u-shaped transformer for image restoration,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2022, pp. 17 683–17 693.
  31. Y. Li, K. Zhang, J. Cao, R. Timofte, and L. Van Gool, “LocalViT: Bringing locality to vision transformers,” arXiv preprint arXiv:2104.05707, 2021.
  32. H. Wu et al., “CvT: Introducing convolutions to vision transformers,” in Proc. IEEE Int. Conf. Comput. Vis., 2021, pp. 22–31.
  33. M. Li, W. Hsu, X. Xie, J. Cong, and W. Gao, “SACNN: Self-attention convolutional neural network for low-dose CT denoising with self-supervised perceptual loss network,” IEEE Trans. Med. Imag., vol. 39, no. 7, pp. 2289–2301, 2020.
  34. Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3D U-Net: learning dense volumetric segmentation from sparse annotation,” in Med. Image Comput. Comput. Assist. Interv.   Springer, 2016, pp. 424–432.
  35. A. Dosovitskiy et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” in Proc. Int. Conf. Learn. Represent., 2020.
  36. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2022, pp. 10 684–10 695.
  37. J. Guo et al., “CMT: Convolutional neural networks meet vision transformers,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2022, pp. 12 175–12 185.
  38. Z. Qiu, T. Yao, and T. Mei, “Learning spatio-temporal representation with pseudo-3D residual networks,” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 5533–5541.
  39. W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Fast and accurate image super-resolution with deep Laplacian pyramid networks,” Proc. IEEE Int. Conf. Comput. Vis., vol. 41, no. 11, pp. 2599–2613, 2018.
  40. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. Image Process., vol. 13, no. 4, pp. 600–612, 2004.
  41. C. H. McCollough et al., “Low-dose CT for the detection and classification of metastatic liver lesions: results of the 2016 low dose CT grand challenge,” Med. Phys., vol. 44, no. 10, pp. e339–e352, 2017.
  42. T. R. Moen et al., “Low-dose CT image and projection dataset,” Med. Phys., vol. 48, no. 2, pp. 902–911, 2021.
  43. L. Yu, M. Shiung, D. Jondal, and C. H. McCollough, “Development and validation of a practical lower-dose-simulation tool for optimizing computed tomography scan protocols,” J. Comput. Assist. Tomogr., vol. 36, no. 4, pp. 477–487, 2012.
  44. N. J. Packard, C. K. Abbey, K. Yang, and J. M. Boone, “Effect of slice thickness on detectability in breast CT using a prewhitened matched filter and simulated mass lesions,” Med. Phys., vol. 39, no. 4, pp. 1818–1830, 2012.
  45. B. N. Narayanan, R. C. Hardie, and T. M. Kebede, “Performance analysis of a computer-aided detection system for lung nodules in CT at different slice thicknesses,” J. Med. Imaging, vol. 5, no. 1, p. 014504, 2018.
  46. I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proc. Int. Conf. Learn. Represent, 2019.
  47. I. Loshchilov and F. Hutter, “SGDR: Stochastic gradient descent with warm restarts,” in Proc. Int. Conf. Learn. Represent, 2017.
  48. P. Goyal et al., “Accurate, large minibatch SGD: Training imagenet in 1 hour,” arXiv preprint arXiv:1706.02677, 2017.
  49. K. C. Chan, S. Zhou, X. Xu, and C. C. Loy, “BasicVSR++: Improving video super-resolution with enhanced propagation and alignment,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2022, pp. 5972–5981.
  50. O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Med. Image Comput. Comput. Assist. Interv.   Springer, 2015, pp. 234–241.
  51. Z. Geng, L. Liang, T. Ding, and I. Zharkov, “RSTT: Real-time spatial temporal transformer for space-time video super-resolution,” in In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2022, pp. 17 441–17 451.
  52. S. Bera and P. K. Biswas, “Noise conscious training of non local neural network powered by self attentive spectral normalized Markovian patch GAN for low dose CT denoising,” IEEE Transactions on Medical Imaging, vol. 40, no. 12, pp. 3663–3673, 2021.
  53. O. Dalmaz, M. Yurt, and T. Çukur, “ResViT: residual vision transformers for multimodal medical image synthesis,” IEEE Trans. Med. Imag., vol. 41, no. 10, pp. 2598–2614, 2022.
  54. Y. Korkmaz, S. U. Dar, M. Yurt, M. Özbey, and T. Cukur, “Unsupervised mri reconstruction via zero-shot learned adversarial transformers,” IEEE Trans. Med. Imag., vol. 41, no. 7, pp. 1747–1763, 2022.
  55. J. Wei, Y. Xia, and Y. Zhang, “M3Net: A multi-model, multi-size, and multi-view deep neural network for brain magnetic resonance image segmentation,” Pattern Recognit., vol. 91, pp. 366–378, 2019.
  56. C. Peng, W.-A. Lin, H. Liao, R. Chellappa, and S. K. Zhou, “SAINT: spatially aware interpolation network for medical slice synthesis,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2020, pp. 7750–7759.
  57. M. Yurt, M. Özbey, S. U. Dar, B. Tinaz, K. K. Oguz, and T. Çukur, “Progressively volumetrized deep generative models for data-efficient contextual learning of mr image recovery,” Med. Image Anal., vol. 78, p. 102429, 2022.
Citations (14)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

GitHub

  1. GitHub - hao1635/LIT-Former (23 stars)