Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Cohort-Individual Cooperative Learning for Multimodal Cancer Survival Analysis (2404.02394v1)

Published 3 Apr 2024 in eess.IV and cs.CV

Abstract: Recently, we have witnessed impressive achievements in cancer survival analysis by integrating multimodal data, e.g., pathology images and genomic profiles. However, the heterogeneity and high dimensionality of these modalities pose significant challenges for extracting discriminative representations while maintaining good generalization. In this paper, we propose a Cohort-individual Cooperative Learning (CCL) framework to advance cancer survival analysis by collaborating knowledge decomposition and cohort guidance. Specifically, first, we propose a Multimodal Knowledge Decomposition (MKD) module to explicitly decompose multimodal knowledge into four distinct components: redundancy, synergy and uniqueness of the two modalities. Such a comprehensive decomposition can enlighten the models to perceive easily overlooked yet important information, facilitating an effective multimodal fusion. Second, we propose a Cohort Guidance Modeling (CGM) to mitigate the risk of overfitting task-irrelevant information. It can promote a more comprehensive and robust understanding of the underlying multimodal data, while avoiding the pitfalls of overfitting and enhancing the generalization ability of the model. By cooperating the knowledge decomposition and cohort guidance methods, we develop a robust multimodal survival analysis model with enhanced discrimination and generalization abilities. Extensive experimental results on five cancer datasets demonstrate the effectiveness of our model in integrating multimodal data for survival analysis.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in Neural Information Processing Systems, vol. 25, 2012.
  2. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.
  3. K. L. Lee, D. B. Pryor, F. E. Harrell Jr, R. M. Califf, V. S. Behar, W. L. Floyd, J. J. Morris, R. A. Waugh, R. E. Whalen, and R. A. Rosati, “Predicting outcome in coronary disease statistical models versus expert clinicians,” The American Journal of Medicine, vol. 80, no. 4, pp. 553–560, 1986.
  4. E. R. Dickson, P. M. Grambsch, T. R. Fleming, L. D. Fisher, and A. Langworthy, “Prognosis in primary biliary cirrhosis: model for decision making,” Hepatology, vol. 10, no. 1, pp. 1–7, 1989.
  5. R. B. D’Agostino, M.-L. Lee, A. J. Belanger, L. A. Cupples, K. Anderson, and W. B. Kannel, “Relation of pooled logistic regression to time dependent cox regression analysis: the framingham heart study,” Statistics in Medicine, vol. 9, no. 12, pp. 1501–1515, 1990.
  6. H. Kappen and J. Neijt, “Neural network analysis to predict treatment outcome,” Annals of Oncology, vol. 4, pp. S31–S34, 1993.
  7. P. Lapuerta, S. P. Azen, and L. LaBree, “Use of neural networks in predicting the risk of coronary artery disease,” Computers and Biomedical Research, vol. 28, no. 1, pp. 38–52, 1995.
  8. L. Ohno-Machado, “A comparison of cox proportional hazards and artificial neural network models for medical prognosis,” Computers in Biology and Medicine, vol. 27, no. 1, pp. 55–65, 1997.
  9. M. B. Eisen, P. T. Spellman, P. O. Brown, and D. Botstein, “Cluster analysis and display of genome-wide expression patterns,” Proceedings of the National Academy of Sciences, vol. 95, no. 25, pp. 14 863–14 868, 1998.
  10. A. A. Alizadeh, M. B. Eisen, R. E. Davis, C. Ma, I. S. Lossos, A. Rosenwald, J. C. Boldrick, H. Sabet, T. Tran, X. Yu et al., “Distinct types of diffuse large b-cell lymphoma identified by gene expression profiling,” Nature, vol. 403, no. 6769, pp. 503–511, 2000.
  11. Y. L. Qiu, H. Zheng, A. Devos, H. Selby, and O. Gevaert, “A meta-learning approach for genomic survival analysis,” Nature Communications, vol. 11, no. 1, p. 6350, 2020.
  12. S. Yousefi, F. Amrollahi, M. Amgad, C. Dong, J. E. Lewis, C. Song, D. A. Gutman, S. H. Halani, J. E. Velazquez Vega, D. J. Brat et al., “Predicting clinical outcomes from large scale cancer genomic profiles with deep survival models,” Scientific Reports, vol. 7, no. 1, p. 11707, 2017.
  13. E. Wulczyn, D. F. Steiner, Z. Xu, A. Sadhwani, H. Wang, I. Flament-Auvigne, C. H. Mermel, P.-H. C. Chen, Y. Liu, and M. C. Stumpe, “Deep learning-based survival prediction for multiple cancer types using histopathology images,” PloS One, vol. 15, no. 6, p. e0233678, 2020.
  14. R. J. Chen, C. Chen, Y. Li, T. Y. Chen, A. D. Trister, R. G. Krishnan, and F. Mahmood, “Scaling vision transformers to gigapixel images via hierarchical self-supervised learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 16 144–16 155.
  15. J. Yao, X. Zhu, J. Jonnagaddala, N. Hawkins, and J. Huang, “Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks,” Medical Image Analysis, vol. 65, p. 101789, 2020.
  16. X. Zhu, J. Yao, F. Zhu, and J. Huang, “Wsisa: Making survival prediction from whole slide histopathological images,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 7234–7242.
  17. G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter, “Self-normalizing neural networks,” Advances in Neural Information Processing Systems, vol. 30, 2017.
  18. W. Shao, J. Liu, Y. Zuo, S. Qi, H. Hong, J. Sheng, Q. Zhu, and D. Zhang, “Fam3l: Feature-aware multi-modal metric learning for integrative survival analysis of human cancers,” IEEE Transactions on Medical Imaging, 2023.
  19. R. Li, X. Wu, A. Li, and M. Wang, “Hfbsurv: hierarchical multimodal fusion with factorized bilinear models for cancer survival prediction,” Bioinformatics, vol. 38, no. 9, pp. 2587–2594, 2022.
  20. N. Braman, J. W. Gordon, E. T. Goossens, C. Willis, M. C. Stumpe, and J. Venkataraman, “Deep orthogonal fusion: multimodal prognostic biomarker discovery integrating radiology, pathology, genomic, and clinical data,” in Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part V 24.   Springer, 2021, pp. 667–677.
  21. R. J. Chen, M. Y. Lu, J. Wang, D. F. Williamson, S. J. Rodig, N. I. Lindeman, and F. Mahmood, “Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis,” IEEE Transactions on Medical Imaging, vol. 41, no. 4, pp. 757–770, 2020.
  22. R. Nakhli, P. A. Moghadam, H. Mi, H. Farahani, A. Baras, B. Gilks, and A. Bashashati, “Sparse multi-modal graph transformer with shared-context processing for representation learning of giga-pixel images,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 11 547–11 557.
  23. Z. Wang, R. Li, M. Wang, and A. Li, “Gpdbn: deep bilinear network integrating both genomic data and pathological images for breast cancer prognosis prediction,” Bioinformatics, vol. 37, no. 18, pp. 2963–2970, 2021.
  24. C. Cui, H. Liu, Q. Liu, R. Deng, Z. Asad, Y. Wang, S. Zhao, H. Yang, B. A. Landman, and Y. Huo, “Survival prediction of brain cancer with incomplete radiology, pathology, genomic, and demographic data,” in International Conference on Medical Image Computing and Computer-Assisted Intervention.   Springer, 2022, pp. 626–635.
  25. R. J. Chen, M. Y. Lu, W.-H. Weng, T. Y. Chen, D. F. Williamson, T. Manz, M. Shady, and F. Mahmood, “Multimodal co-attention transformer for survival prediction in gigapixel whole slide images,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 4015–4025.
  26. G. Jaume, A. Vaidya, R. Chen, D. Williamson, P. Liang, and F. Mahmood, “Modeling dense multimodal interactions between biological pathways and histology for survival prediction,” arXiv preprint arXiv:2304.06819, 2023.
  27. R. J. Chen, M. Y. Lu, D. F. Williamson, T. Y. Chen, J. Lipkova, Z. Noor, M. Shaban, M. Shady, M. Williams, B. Joo et al., “Pan-cancer integrative histology-genomic analysis via multimodal deep learning,” Cancer Cell, vol. 40, no. 8, pp. 865–878, 2022.
  28. Y. Xu and H. Chen, “Multimodal optimal transport-based co-attention transformer with global structure consistency for survival prediction,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, October 2023, pp. 21 241–21 251.
  29. F. Zhou and H. Chen, “Cross-modal translation and alignment for survival analysis,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 21 485–21 494.
  30. Y. Zhang, Y. Xu, J. Chen, F. Xie, and H. Chen, “Prototypical information bottlenecking and disentangling for multimodal cancer survival prediction,” in The International Conference on Learning Representations, 2024.
  31. D. R. Cox, “Regression models and life-tables,” Journal of the Royal Statistical Society: Series B (Methodological), vol. 34, no. 2, pp. 187–202, 1972.
  32. Z. Lv, Y. Lin, R. Yan, Z. Yang, Y. Wang, and F. Zhang, “Pg-tfnet: transformer-based fusion network integrating pathological images and genomic data for cancer survival analysis,” in 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).   IEEE, 2021, pp. 491–496.
  33. J. MacQueen et al., “Some methods for classification and analysis of multivariate observations,” in Proceedings of the fifth Berkeley Symposium on Mathematical Statistics and Probability, vol. 1, no. 14.   Oakland, CA, USA, 1967, pp. 281–297.
  34. H. W. Kuhn, “The hungarian method for the assignment problem,” Naval Research Logistics Quarterly, vol. 2, no. 1-2, pp. 83–97, 1955.
  35. Y. Li, Y. Wang, and Z. Cui, “Decoupled multimodal distilling for emotion recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 6631–6640.
  36. D. Hazarika, R. Zimmermann, and S. Poria, “Misa: Modality-invariant and-specific representations for multimodal sentiment analysis,” in Proceedings of the 28th ACM International Conference on Multimedia, 2020, pp. 1122–1131.
  37. M. Ilse, J. Tomczak, and M. Welling, “Attention-based deep multiple instance learning,” in International Conference on Machine Learning.   PMLR, 2018, pp. 2127–2136.
  38. M. Y. Lu, D. F. Williamson, T. Y. Chen, R. J. Chen, M. Barbieri, and F. Mahmood, “Data-efficient and weakly supervised computational pathology on whole-slide images,” Nature Biomedical Engineering, vol. 5, no. 6, pp. 555–570, 2021.
  39. Z. Shao, H. Bian, Y. Chen, Y. Wang, J. Zhang, X. Ji et al., “Transmil: Transformer based correlated multiple instance learning for whole slide image classification,” Advances in Neural Information Processing Systems, vol. 34, pp. 2136–2147, 2021.
  40. H. Zhang, Y. Meng, Y. Zhao, Y. Qiao, X. Yang, S. E. Coupland, and Y. Zheng, “Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 18 802–18 812.
  41. H. Li, F. Yang, X. Xing, Y. Zhao, J. Zhang, Y. Liu, M. Han, J. Huang, L. Wang, and J. Yao, “Multi-modal multi-instance learning using weakly correlated histopathological images and tabular clinical information,” in Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VIII 24.   Springer, 2021, pp. 529–539.
  42. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in Neural Information Processing Systems, vol. 30, 2017.
  43. G. E. Hinton and S. Roweis, “Stochastic neighbor embedding,” Advances in Neural Information Processing Systems, vol. 15, 2002.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now