Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

SELECTOR: Heterogeneous graph network with convolutional masked autoencoder for multimodal robust prediction of cancer survival (2403.09290v1)

Published 14 Mar 2024 in cs.CV, cs.AI, and cs.LG

Abstract: Accurately predicting the survival rate of cancer patients is crucial for aiding clinicians in planning appropriate treatment, reducing cancer-related medical expenses, and significantly enhancing patients' quality of life. Multimodal prediction of cancer patient survival offers a more comprehensive and precise approach. However, existing methods still grapple with challenges related to missing multimodal data and information interaction within modalities. This paper introduces SELECTOR, a heterogeneous graph-aware network based on convolutional mask encoders for robust multimodal prediction of cancer patient survival. SELECTOR comprises feature edge reconstruction, convolutional mask encoder, feature cross-fusion, and multimodal survival prediction modules. Initially, we construct a multimodal heterogeneous graph and employ the meta-path method for feature edge reconstruction, ensuring comprehensive incorporation of feature information from graph edges and effective embedding of nodes. To mitigate the impact of missing features within the modality on prediction accuracy, we devised a convolutional masked autoencoder (CMAE) to process the heterogeneous graph post-feature reconstruction. Subsequently, the feature cross-fusion module facilitates communication between modalities, ensuring that output features encompass all features of the modality and relevant information from other modalities. Extensive experiments and analysis on six cancer datasets from TCGA demonstrate that our method significantly outperforms state-of-the-art methods in both modality-missing and intra-modality information-confirmed cases. Our codes are made available at https://github.com/panliangrui/Selector.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (62)
  1. Aacr cancer progress report 2023: Advancing the frontiers of cancer science and medicine. Clinical Cancer Research, 29(19):3850–3851, 2023.
  2. Vitamin d3 for reducing mortality from cancer and other outcomes before, during and beyond the covid-19 pandemic: A plea for harvesting low-hanging fruit. Cancer Communications, 42(8):679, 2022.
  3. Multi-head attention mechanism learning for cancer new subtypes and treatment based on cancer multi-omics data. arXiv preprint arXiv:2307.04075, 2023.
  4. Missing data imputation on the 5-year survival prediction of breast cancer patients with unknown discrete values. Computers in biology and medicine, 59:125–133, 2015.
  5. Deep learning in cancer diagnosis, prognosis and treatment selection. Genome Medicine, 13(1):1–17, 2021.
  6. Pacs: Prediction and analysis of cancer subtypes from multi-omics data based on a multi-head attention mechanism model. arXiv preprint arXiv:2308.10917, 2023.
  7. Integrating multiple genomic imaging data for the study of lung metastasis in sarcomas using multi-dimensional constrained joint non-negative matrix factorization. Information Sciences, 576:24–36, 2021.
  8. Artificial intelligence-based multi-omics analysis fuels cancer precision medicine. In Seminars in Cancer Biology, volume 88, pages 187–200. Elsevier, 2023.
  9. Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Medical Image Analysis, 65:101789, 2020.
  10. Big-hypergraph factorization neural network for survival prediction from whole slide image. IEEE Transactions on Image Processing, 31:1149–1160, 2022.
  11. Noise-reducing attention cross fusion learning transformer for histological image classification of osteosarcoma. Biomedical Signal Processing and Control, 77:103824, 2022.
  12. Ldcsf: Local depth convolution-based swim framework for classifying multi-label histopathology images. arXiv preprint arXiv:2308.10446, 2023.
  13. A novel ferroptosis-related gene signature for overall survival prediction in patients with hepatocellular carcinoma. International journal of biological sciences, 16(13):2430, 2020.
  14. Optimizing survival analysis of xgboost for ties to predict disease progression of breast cancer. IEEE Transactions on Biomedical Engineering, 68(1):148–160, 2020.
  15. Risk classification of cancer survival using ann with gene expression data from multiple laboratories. Computers in biology and medicine, 48:1–7, 2014.
  16. Multimodal data fusion for cancer biomarker discovery with deep learning. Nature Machine Intelligence, 5(4):351–362, 2023.
  17. Camr: cross-aligned multimodal representation learning for cancer survival prediction. Bioinformatics, 39(1):btad025, 2023.
  18. Gpdbn: deep bilinear network integrating both genomic data and pathological images for breast cancer prognosis prediction. Bioinformatics, 37(18):2963–2970, 2021.
  19. Hfbsurv: hierarchical multimodal fusion with factorized bilinear models for cancer survival prediction. Bioinformatics, 38(9):2587–2594, 2022.
  20. Deep multimodal graph-based network for survival prediction from highly multiplexed images and patient variables. Computers in Biology and Medicine, 154:106576, 2023.
  21. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering, 5(6):555–570, 2021.
  22. Pan-cancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell, 40(8):865–878, 2022.
  23. Hybrid graph convolutional network with online masked autoencoder for robust multimodal cancer survival prediction. IEEE Transactions on Medical Imaging, 2023.
  24. Modeling dense multimodal interactions between biological pathways and histology for survival prediction. arXiv preprint arXiv:2304.06819, 2023.
  25. Attention-based deep multiple instance learning. In International conference on machine learning, pages 2127–2136. PMLR, 2018.
  26. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4025, 2021.
  27. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in neural information processing systems, 34:2136–2147, 2021.
  28. Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Transactions on Medical Imaging, 41(4):757–770, 2020.
  29. Predicting cancer outcomes from histology and genomics using convolutional networks. Proceedings of the National Academy of Sciences, 115(13):E2970–E2979, 2018.
  30. Long-term cancer survival prediction using multimodal deep learning. Scientific Reports, 11(1):13505, 2021.
  31. Deep learning with multimodal representation for pancancer prognosis prediction. Bioinformatics, 35(14):i446–i454, 2019.
  32. 3d semantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9224–9232, 2018.
  33. Heterogeneous graph masked autoencoders. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 9997–10005, 2023.
  34. Heterogeneous graph neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 793–803, 2019.
  35. Heterogeneous graph structure learning for graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 4697–4705, 2021.
  36. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022.
  37. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096–1103, 2008.
  38. H Holden Thorp. Chatgpt is fun, but not an author, 2023.
  39. Reduce information loss in transformers for pluralistic image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11347–11357, 2022.
  40. Masked autoencoders enable efficient knowledge distillers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24256–24265, 2023.
  41. Contrastive masked autoencoders are stronger vision learners. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  42. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2142–2152, 2023.
  43. Fine-tuning and training of densenet for histopathology image representation using tcga diagnostic slides. Medical Image Analysis, 70:102032, 2021.
  44. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3075–3084, 2019.
  45. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16, pages 574–591. Springer, 2020.
  46. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16133–16142, 2023.
  47. Review the cancer genome atlas (tcga): an immeasurable source of knowledge. Contemporary Oncology/Współczesna Onkologia, 2015(1):68–77, 2015.
  48. Pathway enrichment analysis and visualization of omics data using g: Profiler, gsea, cytoscape and enrichmentmap. Nature protocols, 14(2):482–517, 2019.
  49. Postoperative nomogram for 12-year sarcoma-specific death. Journal of clinical oncology, 20(3):791–796, 2002.
  50. Machine learning for survival analysis: A survey. ACM Computing Surveys (CSUR), 51(6):1–36, 2019.
  51. Survival outcome prediction in cervical cancer: Cox models vs deep-learning model. American journal of obstetrics and gynecology, 220(4):381–e1, 2019.
  52. Xgblc: an improved survival prediction model based on xgboost. Bioinformatics, 38(2):410–418, 2022.
  53. Sarcopenia is a negative prognostic factor after curative resection of colorectal cancer. Annals of surgical oncology, 22:2663–2668, 2015.
  54. Adjusted kaplan–meier estimator and log-rank test with inverse probability of treatment weighting for survival data. Statistics in medicine, 24(20):3089–3110, 2005.
  55. Learning and reasoning with the graph structure representation in robotic surgery. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part III 23, pages 627–636. Springer, 2020.
  56. Pooling regularized graph neural network for fmri biomarker analysis. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part VII 23, pages 625–635. Springer, 2020.
  57. Cell graph neural networks enable the precise prediction of patient survival in gastric cancer. NPJ precision oncology, 6(1):45, 2022.
  58. Ectopic gene conversions in the human genome. Genomics, 93(1):27–32, 2009.
  59. Gps-net: Graph property sensing network for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3746–3753, 2020.
  60. A spatiotemporal graph neural network for session-based recommendation. Expert Systems with Applications, 202:117114, 2022.
  61. Convmae: Masked convolution meets masked autoencoders. arXiv preprint arXiv:2205.03892, 2022.
  62. Denoising masked autoencoders help robust classification. In The Eleventh International Conference on Learning Representations, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Liangrui Pan (20 papers)
  2. Yijun Peng (3 papers)
  3. Yan Li (505 papers)
  4. Xiang Wang (279 papers)
  5. Wenjuan Liu (17 papers)
  6. Liwen Xu (11 papers)
  7. Qingchun Liang (7 papers)
  8. Shaoliang Peng (21 papers)
Citations (1)
View on Semantic Scholar