Cross-modality Attention-based Multimodal Fusion for Non-small Cell Lung Cancer (NSCLC) Patient Survival Prediction (2308.09831v2)
Abstract: Cancer prognosis and survival outcome predictions are crucial for therapeutic response estimation and for stratifying patients into various treatment groups. Medical domains concerned with cancer prognosis are abundant with multiple modalities, including pathological image data and non-image data such as genomic information. To date, multimodal learning has shown potential to enhance clinical prediction model performance by extracting and aggregating information from different modalities of the same subject. This approach could outperform single modality learning, thus improving computer-aided diagnosis and prognosis in numerous medical applications. In this work, we propose a cross-modality attention-based multimodal fusion pipeline designed to integrate modality-specific knowledge for patient survival prediction in non-small cell lung cancer (NSCLC). Instead of merely concatenating or summing up the features from different modalities, our method gauges the importance of each modality for feature fusion with cross-modality relationship when infusing the multimodal features. Compared with single modality, which achieved c-index of 0.5772 and 0.5885 using solely tissue image data or RNA-seq data, respectively, the proposed fusion approach achieved c-index 0.6587 in our experiment, showcasing the capability of assimilating modality-specific knowledge from varied modalities.
- Mobadersany, P., Yousefi, S., Amgad, M., Gutman, D. A., Barnholtz-Sloan, J. S., Velázquez Vega, J. E., Brat, D. J., and Cooper, L. A., “Predicting cancer outcomes from histology and genomics using convolutional networks,” Proceedings of the National Academy of Sciences 115(13), E2970–E2979 (2018).
- Cheerla, A. and Gevaert, O., “Deep learning with multimodal representation for pancancer prognosis prediction,” Bioinformatics 35(14), i446–i454 (2019).
- Chen, R. J., Lu, M. Y., Weng, W.-H., Chen, T. Y., Williamson, D. F., Manz, T., Shady, M., and Mahmood, F., “Multimodal co-attention transformer for survival prediction in gigapixel whole slide images,” in [Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) ], 4015–4025 (October 2021).
- Ilse, M., Tomczak, J., and Welling, M., “Attention-based deep multiple instance learning,” in [International conference on machine learning ], 2127–2136, PMLR (2018).
- Jaume, G., Vaidya, A., Chen, R., Williamson, D., Liang, P., and Mahmood, F., “Modeling dense multimodal interactions between biological pathways and histology for survival prediction,” arXiv preprint arXiv:2304.06819 (2023).
- Chen, R. J., Lu, M. Y., Williamson, D. F., Chen, T. Y., Lipkova, J., Noor, Z., Shaban, M., Shady, M., Williams, M., Joo, B., et al., “Pan-cancer integrative histology-genomic analysis via multimodal deep learning,” Cancer Cell 40(8), 865–878 (2022).
- Klambauer, G., Unterthiner, T., Mayr, A., and Hochreiter, S., “Self-normalizing neural networks,” Advances in neural information processing systems 30 (2017).
- Deng, R., Cui, C., Remedios, L. W., Bao, S., Womick, R. M., Chiron, S., Li, J., Roland, J. T., Lau, K. S., Liu, Q., et al., “Cross-scale attention guided multi-instance learning for crohn’s disease diagnosis with pathological images,” in [International Workshop on Multiscale Multimodal Medical Imaging ], 24–33, Springer (2022).
- Socinski, M. A., Jotte, R. M., Cappuzzo, F., Orlandi, F., Stroyakovskiy, D., Nogami, N., Rodríguez-Abreu, D., Moro-Sibilot, D., Thomas, C. A., Barlesi, F., et al., “Atezolizumab for first-line treatment of metastatic nonsquamous nsclc,” New England Journal of Medicine 378(24), 2288–2301 (2018).
- Lu, M. Y., Williamson, D. F., Chen, T. Y., Chen, R. J., Barbieri, M., and Mahmood, F., “Data-efficient and weakly supervised computational pathology on whole-slide images,” Nature Biomedical Engineering 5(6), 555–570 (2021).
- Wu, Y., “Elastic net for cox’s proportional hazards model with a solution path algorithm,” Statistica Sinica 22, 27 (2012).
- Yao, J., Zhu, X., Jonnagaddala, J., Hawkins, N., and Huang, J., “Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks,” Medical Image Analysis 65, 101789 (2020).
- Ruining Deng (66 papers)
- Nazim Shaikh (4 papers)
- Gareth Shannon (1 paper)
- Yao Nie (11 papers)