Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Super-Resolution Based Patch-Free 3D Image Segmentation with High-Frequency Guidance (2210.14645v2)

Published 26 Oct 2022 in eess.IV and cs.CV

Abstract: High resolution (HR) 3D images are widely used nowadays, such as medical images like Magnetic Resonance Imaging (MRI) and Computed Tomography (CT). However, segmentation of these 3D images remains a challenge due to their high spatial resolution and dimensionality in contrast to currently limited GPU memory. Therefore, most existing 3D image segmentation methods use patch-based models, which have low inference efficiency and ignore global contextual information. To address these problems, we propose a super-resolution (SR) based patch-free 3D image segmentation framework that can realize HR segmentation from a global-wise low-resolution (LR) input. The framework contains two sub-tasks, of which semantic segmentation is the main task and super resolution is an auxiliary task aiding in rebuilding the high frequency information from the LR input. To furthermore balance the information loss with the LR input, we propose a High-Frequency Guidance Module (HGM), and design an efficient selective cropping algorithm to crop an HR patch from the original image as restoration guidance for it. In addition, we also propose a Task-Fusion Module (TFM) to exploit the inter connections between segmentation and SR task, realizing joint optimization of the two tasks. When predicting, only the main segmentation task is needed, while other modules can be removed for acceleration. The experimental results on two different datasets show that our framework has a four times higher inference speed compared to traditional patch-based methods, while its performance also surpasses other patch-based and patch-free models.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Y. Tang, Y. Tang, Y. Zhu, J. Xiao, and R. M. Summers, “E2net: An edge enhanced network for accurate liver and tumor segmentation on ct scans,” in International Conference on Medical Image Computing and Computer-Assisted Intervention.   Springer, 2020, pp. 512–522.
  2. H. Huang, L. Lin, R. Tong, H. Hu, Q. Zhang, Y. Iwamoto, X. Han, Y.-W. Chen, and J. Wu, “Unet 3+: A full-scale connected unet for medical image segmentation,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).   IEEE, 2020, pp. 1055–1059.
  3. H. Huang, H. Zheng, L. Lin, M. Cai, H. Hu, Q. Zhang, Q. Chen, Y. Iwamoto, X. Han, Y.-W. Chena et al., “Medical image segmentation with deep atlas prior,” IEEE Transactions on Medical Imaging, 2021.
  4. Y.-H. Wu, S.-H. Gao, J. Mei, J. Xu, D.-P. Fan, R.-G. Zhang, and M.-M. Cheng, “Jcs: An explainable covid-19 diagnosis system by joint classification and segmentation,” IEEE Transactions on Image Processing, vol. 30, pp. 3113–3126, 2021.
  5. Q. Shao, L. Gong, K. Ma, H. Liu, and Y. Zheng, “Attentive ct lesion detection using deep pyramid inference with multi-scale booster,” in International Conference on Medical Image Computing and Computer-Assisted Intervention.   Springer, 2019, pp. 301–309.
  6. M. Zlocha, Q. Dou, and B. Glocker, “Improving retinanet for ct lesion detection with dense masks from weak recist labels,” in International Conference on Medical Image Computing and Computer-Assisted Intervention.   Springer, 2019, pp. 402–410.
  7. S. Kim, S. An, P. Chikontwe, and S. H. Park, “Bidirectional rnn-based few shot learning for 3d medical image segmentation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 3, 2021, pp. 1808–1816.
  8. Y. Wang, Y. Zhou, W. Shen, S. Park, E. K. Fishman, and A. L. Yuille, “Abdominal multi-organ segmentation with organ-attention networks and statistical fusion,” Medical Image Analysis, vol. 55, pp. 88–102, 2019.
  9. Y. Xia, D. Yang, Z. Yu, F. Liu, J. Cai, L. Yu, Z. Zhu, D. Xu, A. Yuille, and H. Roth, “Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptation,” Medical Image Analysis, vol. 65, p. 101766, 2020.
  10. T. Kitrungrotsakul, X.-H. Han, Y. Iwamoto, L. Lin, A. H. Foruzan, W. Xiong, and Y.-W. Chen, “Vesselnet: A deep convolutional neural network with multi pathways for robust hepatic vessel segmentation,” Computerized Medical Imaging and Graphics, vol. 75, pp. 74–83, 2019.
  11. F. Madesta, R. Schmitz, T. Rösch, and R. Werner, “Widening the focus: Biomedical image segmentation challenges and the underestimated role of patch sampling and inference strategies,” in International Conference on Medical Image Computing and Computer-Assisted Intervention.   Springer, 2020, pp. 289–298.
  12. Y. Tang, R. Gao, H. H. Lee, S. Han, Y. Chen, D. Gao, V. Nath, C. Bermudez, M. R. Savona, R. G. Abramson et al., “High-resolution 3d abdominal segmentation with random patch network fusion,” Medical Image Analysis, vol. 69, p. 101894, 2021.
  13. H. Yang, C. Shan, A. Bouwman, A. F. Kolen, and P. H. de With, “Efficient and robust instrument segmentation in 3d ultrasound using patch-of-interest-fusenet with hybrid loss,” Medical Image Analysis, vol. 67, p. 101842, 2021.
  14. Y. Huang, L. Shao, and A. F. Frangi, “Simultaneous super-resolution and cross-modality synthesis of 3d medical images using weakly-supervised joint convolutional sparse coding,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 6070–6079.
  15. G. Zeng and G. Zheng, “Holistic decomposition convolution for effective semantic segmentation of medical volume images,” Medical Image Analysis, vol. 57, pp. 149–164, 2019.
  16. L. Wang, D. Li, Y. Zhu, L. Tian, and Y. Shan, “Dual super-resolution learning for semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 3774–3783.
  17. H. Wang, L. Lin, H. Hu, Q. Chen, Y. Li, Y. Iwamoto, X.-H. Han, Y.-W. Chen, and R. Tong, “Patch-free 3d medical image segmentation driven by super-resolution technique and self-supervised guidance,” in International Conference on Medical Image Computing and Computer-Assisted Intervention.   Springer, 2021, pp. 131–141.
  18. O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention.   Springer, 2015, pp. 234–241.
  19. O. Oktay, J. Schlemper, L. L. Folgoc, M. Lee, M. Heinrich, K. Misawa, K. Mori, S. McDonagh, N. Y. Hammerla, B. Kainz et al., “Attention u-net: Learning where to look for the pancreas,” arXiv preprint arXiv:1804.03999, 2018.
  20. L. Zhou, C. Zhang, and M. Wu, “D-linknet: Linknet with pretrained encoder and dilated convolution for high resolution satellite imagery road extraction,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018, pp. 182–186.
  21. Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3d u-net: learning dense volumetric segmentation from sparse annotation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention.   Springer, 2016, pp. 424–432.
  22. F. Milletari, N. Navab, and S.-A. Ahmadi, “V-net: Fully convolutional neural networks for volumetric medical image segmentation,” in 2016 Fourth International Conference on 3D Vision (3DV).   IEEE, 2016, pp. 565–571.
  23. A. Hatamizadeh, V. Nath, Y. Tang, D. Yang, H. Roth, and D. Xu, “Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images,” arXiv preprint arXiv:2201.01266, 2022.
  24. Y. Tang, D. Yang, W. Li, H. R. Roth, B. Landman, D. Xu, V. Nath, and A. Hatamizadeh, “Self-supervised pre-training of swin transformers for 3d medical image analysis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 20 730–20 740.
  25. H. Kim, J. Jung, J. Kim, B. Cho, J. Kwak, J. Y. Jang, S.-w. Lee, J.-G. Lee, and S. M. Yoon, “Abdominal multi-organ auto-segmentation using 3d-patch-based deep convolutional neural network,” Scientific Reports, vol. 10, no. 1, pp. 1–9, 2020.
  26. P.-Y. Kao, F. Shailja, J. Jiang, A. Zhang, A. Khan, J. W. Chen, and B. Manjunath, “Improving patch-based convolutional neural networks for mri brain tumor segmentation by leveraging location information,” Frontiers in Neuroscience, vol. 13, p. 1449, 2020.
  27. F. Lekien and J. Marsden, “Tricubic interpolation in three dimensions,” International Journal for Numerical Methods in Engineering, vol. 63, no. 3, pp. 455–471, 2005.
  28. W. Shi, J. Caballero, F. Huszar, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 1874–1883.
  29. N.-V. Ho, T. Nguyen, G.-H. Diep, N. Le, and B.-S. Hua, “Point-unet: A context-aware point-based neural network for volumetric segmentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention.   Springer, 2021, pp. 644–655.
  30. A. Myronenko, “3d mri brain tumor segmentation using autoencoder regularization,” in Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 4th International Workshop, BrainLes 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Revised Selected Papers, Part II 4.   Springer, 2019, pp. 311–320.
  31. L. Yu, X. Yang, H. Chen, J. Qin, and P. A. Heng, “Volumetric convnets with mixed residual connections for automated prostate segmentation from 3d mr images,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31, no. 1, 2017.
  32. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.
  33. X. Wang, X. Jiang, H. Ding, and J. Liu, “Bi-directional dermoscopic feature learning and multi-scale consistent decision fusion for skin lesion segmentation,” IEEE transactions on image processing, vol. 29, pp. 3039–3051, 2019.
  34. J. Fu, J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, and H. Lu, “Dual attention network for scene segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 3146–3154.
  35. S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J. S. Kirby, J. B. Freymann, K. Farahani, and C. Davatzikos, “Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features,” Scientific Data, vol. 4, no. 1, pp. 1–13, 2017.
  36. S. Bakas, M. Reyes, A. Jakab, S. Bauer, M. Rempfler, A. Crimi, R. T. Shinohara, C. Berger, S. M. Ha, M. Rozycki et al., “Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge,” arXiv preprint arXiv:1811.02629, 2018.
  37. B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, Y. Burren, N. Porz, J. Slotboom, R. Wiest et al., “The multimodal brain tumor image segmentation benchmark (brats),” IEEE Transactions on Medical Imaging, vol. 34, no. 10, pp. 1993–2024, 2014.
  38. F. Isensee, P. Kickingereder, W. Wick, M. Bendszus, and K. H. Maier-Hein, “No new-net,” in International MICCAI Brainlesion Workshop.   Springer, 2018, pp. 234–244.
  39. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in International Conference on Learning Representations, 2015.
  40. A. Hatamizadeh, Y. Tang, V. Nath, D. Yang, A. Myronenko, B. Landman, H. R. Roth, and D. Xu, “Unetr: Transformers for 3d medical image segmentation,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp. 574–584.
  41. B. Alsallakh, N. Kokhlikyan, V. Miglani, J. Yuan, and O. Reblitz-Richardson, “Mind the pad – cnns can develop blind spots,” in International Conference on Learning Representations, 2021. [Online]. Available: \colorbluehttps://openreview.net/forum?id=m1CD7tPubNy
  42. M.-W. Shao, L. Li, D.-Y. Meng, and W.-M. Zuo, “Uncertainty guided multi-scale attention network for raindrop removal from a single image,” IEEE Transactions on Image Processing, vol. 30, pp. 4828–4839, 2021.
  43. Y. Huang, F. Zheng, R. Cong, W. Huang, M. R. Scott, and L. Shao, “Mcmt-gan: Multi-task coherent modality transferable gan for 3d brain image synthesis,” IEEE Transactions on Image Processing, vol. 29, pp. 8187–8198, 2020.
  44. Y. Li, Y. Iwamoto, L. Lin, R. Xu, R. Tong, and Y.-W. Chen, “Volumenet: A lightweight parallel network for super-resolution of mr and ct volumetric data,” IEEE Transactions on Image Processing, vol. 30, pp. 4840–4854, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (9)
  1. Hongyi Wang (62 papers)
  2. Lanfen Lin (36 papers)
  3. Hongjie Hu (6 papers)
  4. Qingqing Chen (9 papers)
  5. Yinhao Li (19 papers)
  6. Yutaro Iwamoto (12 papers)
  7. Xian-Hua Han (6 papers)
  8. Yen-Wei Chen (36 papers)
  9. Ruofeng Tong (25 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now