Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Cross-modality Guidance-aided Multi-modal Learning with Dual Attention for MRI Brain Tumor Grading (2401.09029v1)

Published 17 Jan 2024 in cs.CV and cs.AI

Abstract: Brain tumor represents one of the most fatal cancers around the world, and is very common in children and the elderly. Accurate identification of the type and grade of tumor in the early stages plays an important role in choosing a precise treatment plan. The Magnetic Resonance Imaging (MRI) protocols of different sequences provide clinicians with important contradictory information to identify tumor regions. However, manual assessment is time-consuming and error-prone due to big amount of data and the diversity of brain tumor types. Hence, there is an unmet need for MRI automated brain tumor diagnosis. We observe that the predictive capability of uni-modality models is limited and their performance varies widely across modalities, and the commonly used modality fusion methods would introduce potential noise, which results in significant performance degradation. To overcome these challenges, we propose a novel cross-modality guidance-aided multi-modal learning with dual attention for addressing the task of MRI brain tumor grading. To balance the tradeoff between model efficiency and efficacy, we employ ResNet Mix Convolution as the backbone network for feature extraction. Besides, dual attention is applied to capture the semantic interdependencies in spatial and slice dimensions respectively. To facilitate information interaction among modalities, we design a cross-modality guidance-aided module where the primary modality guides the other secondary modalities during the process of training, which can effectively leverage the complementary information of different MRI modalities and meanwhile alleviate the impact of the possible noise.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. P. Yang, Y. Wang, X. Peng, G. You, W. Zhang, W. Yan, Z. Bao, Y. Wang, X. Qiu, and T. Jiang, “Management and survival rates in patients with glioma in china (2004–2010): a retrospective study from a single-institution,” Journal of neuro-oncology, vol. 113, no. 2, pp. 259–266, 2013.
  2. C. Krafft, S. B. Sobottka, G. Schackert, and R. Salzer, “Analysis of human brain tissue, brain tumors and tumor cells by infrared spectroscopic mapping,” Analyst, vol. 129, no. 10, pp. 921–925, 2004.
  3. S. Lefkovits, L. Lefkovits, and L. Szilágyi, “Hgg and lgg brain tumor segmentation in multi-modal mri using pretrained convolutional neural networks of amazon sagemaker,” Applied Sciences, vol. 12, no. 7, p. 3620, 2022.
  4. M. E. Laino, R. Young, K. Beal, S. Haque, Y. Mazaheri, G. Corrias, A. G. Bitencourt, S. Karimi, and S. B. Thakur, “Magnetic resonance spectroscopic imaging in gliomas: clinical diagnosis and radiotherapy planning,” BJR— Open, vol. 2, no. 1, p. 20190026, 2020.
  5. A. K. Altwairgi, S. Raja, M. Manzoor, S. Aldandan, E. Alsaeed, A. Balbaid, H. Alhussain, Y. Orz, A. Lary, and A. A. Alsharm, “Management and treatment recommendations for world health organization grade iii and iv gliomas,” International Journal of Health Sciences, vol. 11, no. 3, p. 54, 2017.
  6. N. A. Oberheim Bush and S. Chang, “Treatment strategies for low-grade glioma in adults,” Journal of oncology practice, vol. 12, no. 12, pp. 1235–1241, 2016.
  7. W. Aiman and A. Rayi, “Low grade gliomas,” StatPearls [Internet], 2022.
  8. D. A. Forst, B. V. Nahed, J. S. Loeffler, and T. T. Batchelor, “Low-grade gliomas,” The oncologist, vol. 19, no. 4, pp. 403–413, 2014.
  9. T. Reithmeier, W. Lopez, S. Doostkam, M. Machein, M. Pinsker, M. Trippel, and G. Nikkhah, “Intraindividual comparison of histopathological diagnosis obtained by stereotactic serial biopsy to open surgical resection specimen in patients with intracranial tumours,” Clinical Neurology and Neurosurgery, vol. 115, no. 10, pp. 1955–1960, 2013.
  10. Y. Mizobuchi, K. Nakajima, T. Fujihara, K. Matsuzaki, H. Mure, S. Nagahiro, and Y. Takagi, “The risk of hemorrhage in stereotactic biopsy for brain tumors,” The Journal of Medical Investigation, vol. 66, no. 3.4, pp. 314–318, 2019.
  11. S. Bauer, R. Wiest, L.-P. Nolte, and M. Reyes, “A survey of mri-based medical image analysis for brain tumor studies,” Physics in Medicine & Biology, vol. 58, no. 13, p. R97, 2013.
  12. J. Juan-Albarracín, E. Fuster-Garcia, J. V. Manjon, M. Robles, F. Aparici, L. Martí-Bonmatí, and J. M. Garcia-Gomez, “Automated glioblastoma segmentation based on a multiparametric structured unsupervised classification,” PloS one, vol. 10, no. 5, p. e0125143, 2015.
  13. H.-h. Cho and H. Park, “Classification of low-grade and high-grade glioma using multi-modal image radiomics features,” in 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).   IEEE, 2017, pp. 3081–3084.
  14. J. Cheng, J. Liu, H. Yue, H. Bai, Y. Pan, and J. Wang, “Prediction of glioma grade using intratumoral and peritumoral radiomic features from multiparametric mri images,” IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2020.
  15. S. Liang, R. Zhang, D. Liang, T. Song, T. Ai, C. Xia, L. Xia, and Y. Wang, “Multimodal 3d densenet for idh genotype prediction in gliomas,” Genes, vol. 9, no. 8, p. 382, 2018.
  16. H. Ouerghi, O. Mourali, and E. Zagrouba, “Glioma classification via mr images radiomics analysis,” The Visual Computer, vol. 38, no. 4, pp. 1427–1441, 2022.
  17. Y. Zhuge, H. Ning, P. Mathen, J. Y. Cheng, A. V. Krauze, K. Camphausen, and R. W. Miller, “Automated glioma grading on conventional mri images using deep convolutional neural networks,” Medical physics, vol. 47, no. 7, pp. 3044–3053, 2020.
  18. I. Shahzadi, T. B. Tang, F. Meriadeau, and A. Quyyum, “Cnn-lstm: cascaded framework for brain tumour classification,” in 2018 IEEE-EMBS Conference on Biomedical Engineering and Sciences (IECBES).   IEEE, 2018, pp. 633–637.
  19. S. Mai, Y. Zeng, and H. Hu, “Multimodal information bottleneck: Learning minimal sufficient unimodal and multimodal representations,” IEEE Transactions on Multimedia, 2022.
  20. P. Maneesha, T. Singh, R. Nayar, and S. Kumar, “Multi modal medical image fusion using convolution neural network,” in 2019 Third International Conference on Inventive Systems and Control (ICISC).   IEEE, 2019, pp. 351–357.
  21. L. Pei, L. Vidyaratne, W.-W. Hsu, M. M. Rahman, and K. M. Iftekharuddin, “Brain tumor classification using 3d convolutional neural network,” in International MICCAI brainlesion workshop.   Springer, 2020, pp. 335–342.
  22. J. Cheng, M. Gao, J. Liu, H. Yue, H. Kuang, J. Liu, and J. Wang, “Multimodal disentangled variational autoencoder with game theoretic interpretability for glioma grading,” IEEE Journal of Biomedical and Health Informatics, vol. 26, no. 2, pp. 673–684, 2021.
  23. Z. Ning, J. Luo, Q. Xiao, L. Cai, Y. Chen, X. Yu, J. Wang, and Y. Zhang, “Multi-modal magnetic resonance imaging-based grading analysis for gliomas by integrating radiomics and deep features,” Annals of Translational Medicine, vol. 9, no. 4, 2021.
  24. J. Seetha and S. S. Raja, “Brain tumor classification using convolutional neural networks,” Biomedical & Pharmacology Journal, vol. 11, no. 3, p. 1457, 2018.
  25. S. Deepak and P. Ameer, “Brain tumor classification using deep cnn features via transfer learning,” Computers in biology and medicine, vol. 111, p. 103345, 2019.
  26. F. J. Díaz-Pernas, M. Martínez-Zarzuela, M. Antón-Rodríguez, and D. González-Ortega, “A deep learning approach for brain tumor classification and segmentation using a multiscale convolutional neural network,” in Healthcare, vol. 9, no. 2.   MDPI, 2021, p. 153.
  27. K. Kaplan, Y. Kaya, M. Kuncan, and H. M. Ertunç, “Brain tumor classification using modified local binary patterns (lbp) feature extraction methods,” Medical hypotheses, vol. 139, p. 109696, 2020.
  28. T. K. Dutta and D. R. Nayak, “Cdanet: Channel split dual attention based cnn for brain tumor classification in mr images,” in 2022 IEEE International Conference on Image Processing (ICIP).   IEEE, 2022, pp. 4208–4212.
  29. J. Amin, M. Sharif, N. Gul, M. Yasmin, and S. A. Shad, “Brain tumor classification based on dwt fusion of mri sequences using convolutional neural network,” Pattern Recognition Letters, vol. 129, pp. 115–122, 2020.
  30. M. Kaur and D. Singh, “Multi-modality medical image fusion technique using multi-objective differential evolution based deep neural networks,” Journal of Ambient Intelligence and Humanized Computing, vol. 12, no. 2, pp. 2483–2493, 2021.
  31. W. Tan, P. Tiwari, H. M. Pandey, C. Moreira, and A. K. Jaiswal, “Multimodal medical image fusion algorithm in the era of big data,” Neural Computing and Applications, pp. 1–21, 2020.
  32. A. Wang, X. Luo, Z. Zhang, and X.-J. Wu, “A disentangled representation based brain image fusion via group lasso penalty,” Frontiers in Neuroscience, vol. 16, 2022.
  33. M. M. Almasri and A. M. Alajlan, “Artificial intelligence-based multimodal medical image fusion using hybrid s2 optimal cnn,” Electronics, vol. 11, no. 14, p. 2124, 2022.
  34. H. Hermessi, O. Mourali, and E. Zagrouba, “Multimodal medical image fusion review: Theoretical background and recent advances,” Signal Processing, vol. 183, p. 108036, 2021.
  35. Y. Liu, Q. Fan, S. Zhang, H. Dong, T. Funkhouser, and L. Yi, “Contrastive multimodal fusion with tupleinfonce,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 754–763.
  36. J. Hu, Y. Liu, J. Zhao, and Q. Jin, “Mmgcn: Multimodal fusion via deep graph convolution network for emotion recognition in conversation,” arXiv preprint arXiv:2107.06779, 2021.
  37. Y. Wu, P. Zhan, Y. Zhang, L. Wang, and Z. Xu, “Multimodal fusion with co-attention networks for fake news detection,” in Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 2021, pp. 2560–2569.
  38. Y. Zhang, N. He, J. Yang, Y. Li, D. Wei, Y. Huang, Y. Zhang, Z. He, and Y. Zheng, “mmformer: Multimodal medical transformer for incomplete multimodal learning of brain tumor segmentation,” arXiv preprint arXiv:2206.02425, 2022.
  39. T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” in Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 2016, pp. 785–794.
  40. Z. Han, F. Yang, J. Huang, C. Zhang, and J. Yao, “Multimodal dynamics: Dynamical fusion for trustworthy multimodal classification,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 20 707–20 717.
  41. W. Wang, D. Tran, and M. Feiszli, “What makes training multi-modal classification networks hard?” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 12 695–12 705.
  42. S. Chatterjee, F. A. Nizamani, A. Nürnberger, and O. Speck, “Classification of brain tumours in mr images using deep spatiospatial models,” Scientific Reports, vol. 12, no. 1, pp. 1–11, 2022.
  43. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
  44. J. Fu, J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, and H. Lu, “Dual attention network for scene segmentation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 3146–3154.
  45. B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, Y. Burren, N. Porz, J. Slotboom, R. Wiest et al., “The multimodal brain tumor image segmentation benchmark (brats),” IEEE transactions on medical imaging, vol. 34, no. 10, pp. 1993–2024, 2014.
  46. S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J. S. Kirby, J. B. Freymann, K. Farahani, and C. Davatzikos, “Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features,” Scientific data, vol. 4, no. 1, pp. 1–13, 2017.
  47. S. Bakas, M. Reyes, A. Jakab, S. Bauer, M. Rempfler, A. Crimi, R. T. Shinohara, C. Berger, S. M. Ha, M. Rozycki et al., “Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge,” arXiv preprint arXiv:1811.02629, 2018.
  48. B. C. Lowekamp, D. T. Chen, L. Ibáñez, and D. Blezek, “The design of simpleitk,” Frontiers in neuroinformatics, vol. 7, p. 45, 2013.
  49. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in International Conference on Learning Representations (ICLR), 2015.
  50. J. Amin, M. Sharif, M. Yasmin, T. Saba, M. A. Anjum, and S. L. Fernandes, “A new approach for brain tumor segmentation and classification based on score level fusion using transfer learning,” Journal of medical systems, vol. 43, no. 11, pp. 1–16, 2019.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets