MOAB: Multi-Modal Outer Arithmetic Block For Fusion Of Histopathological Images And Genetic Data For Brain Tumor Grading (2403.06349v1)
Abstract: Brain tumors are an abnormal growth of cells in the brain. They can be classified into distinct grades based on their growth. Often grading is performed based on a histological image and is one of the most significant predictors of a patients prognosis, the higher the grade, the more aggressive the tumor. Correct diagnosis of a tumor grade remains challenging. Though histopathological grading has been shown to be prognostic, results are subject to interobserver variability, even among experienced pathologists. Recently, the World Health Organization reported that advances in molecular genetics have led to improvements in tumor classification. This paper seeks to integrate histological images and genetic data for improved computer-aided diagnosis. We propose a novel Multi-modal Outer Arithmetic Block (MOAB) based on arithmetic operations to combine latent representations of the different modalities for predicting the tumor grade (Grade \rom{2}, \rom{3} and \rom{4}). Extensive experiments evaluate the effectiveness of our approach. By applying MOAB to The Cancer Genome Atlas (TCGA) glioma dataset, we show that it can improve separation between similar classes (Grade \rom{2} and \rom{3}) and outperform prior state-of-the-art grade classification techniques.
- “Visually meaningful histopathological features for automatic grading of prostate cancer,” IEEE journal of biomedical and health informatics, vol. 21, no. 4, pp. 1027–1038, 2016.
- “Automated grading of breast cancer histopathology using cascaded ensemble with combination of multi-level image features,” Neurocomputing, vol. 229, pp. 34–44, 2017.
- “Pan-renal cell carcinoma classification and survival prediction from histopathology images using deep learning,” Scientific reports, vol. 9, no. 1, pp. 1–9, 2019.
- “Salmon: survival analysis learning with multi-omics neural networks on breast cancer,” Frontiers in genetics, vol. 10, pp. 166, 2019.
- “Deep learning-based breast cancer grading and survival analysis on whole-slide histopathology images,” Scientific reports, vol. 12, no. 1, pp. 1–12, 2022.
- “Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis,” IEEE Transactions on Medical Imaging, 2020.
- “Multi-modal classification for human breast cancer prognosis prediction: proposal of deep-learning based stacked ensemble model,” IEEE/ACM transactions on computational biology and bioinformatics, 2020.
- “Richer fusion network for breast cancer classification based on multimodal data,” BMC Medical Informatics and Decision Making, vol. 21, no. 1, pp. 1–15, 2021.
- “Deep orthogonal fusion: Multimodal prognostic biomarker discovery integrating radiology, pathology, genomic, and clinical data,” in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2021, pp. 667–677.
- “A multi-modal fusion framework based on multi-task correlation learning for cancer prognosis prediction,” Artificial Intelligence in Medicine, vol. 126, pp. 102260, 2022.
- “Predicting cancer outcomes from histology and genomics using convolutional networks,” Proceedings of the National Academy of Sciences, vol. 115, no. 13, pp. E2970–E2979, 2018.
- “Glioblastoma multiforme: a review of its epidemiology and pathogenesis through clinical presentation and treatment,” Asian Pacific journal of cancer prevention: APJCP, vol. 18, no. 1, pp. 3, 2017.
- “A convnet for the 2020s,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 11976–11986.
- “Convunext: An efficient convolution neural network for medical image segmentation,” Knowledge-Based Systems, vol. 253, pp. 109512, 2022.
- Laurens Van der Maaten and Geoffrey Hinton, “Visualizing data using t-sne.,” Journal of machine learning research, vol. 9, no. 11, 2008.
- “A re-examination of text categorization methods,” in Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, 1999, pp. 42–49.