Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

iMD4GC: Incomplete Multimodal Data Integration to Advance Precise Treatment Response Prediction and Survival Analysis for Gastric Cancer (2404.01192v1)

Published 1 Apr 2024 in eess.IV and cs.CV

Abstract: Gastric cancer (GC) is a prevalent malignancy worldwide, ranking as the fifth most common cancer with over 1 million new cases and 700 thousand deaths in 2020. Locally advanced gastric cancer (LAGC) accounts for approximately two-thirds of GC diagnoses, and neoadjuvant chemotherapy (NACT) has emerged as the standard treatment for LAGC. However, the effectiveness of NACT varies significantly among patients, with a considerable subset displaying treatment resistance. Ineffective NACT not only leads to adverse effects but also misses the optimal therapeutic window, resulting in lower survival rate. However, existing multimodal learning methods assume the availability of all modalities for each patient, which does not align with the reality of clinical practice. The limited availability of modalities for each patient would cause information loss, adversely affecting predictive accuracy. In this study, we propose an incomplete multimodal data integration framework for GC (iMD4GC) to address the challenges posed by incomplete multimodal data, enabling precise response prediction and survival analysis. Specifically, iMD4GC incorporates unimodal attention layers for each modality to capture intra-modal information. Subsequently, the cross-modal interaction layers explore potential inter-modal interactions and capture complementary information across modalities, thereby enabling information compensation for missing modalities. To evaluate iMD4GC, we collected three multimodal datasets for GC study: GastricRes (698 cases) for response prediction, GastricSur (801 cases) for survival analysis, and TCGA-STAD (400 cases) for survival analysis. The scale of our datasets is significantly larger than previous studies. The iMD4GC achieved impressive performance with an 80.2% AUC on GastricRes, 71.4% C-index on GastricSur, and 66.1% C-index on TCGA-STAD, significantly surpassing other compared methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. Burden of gastric cancer. Clinical Gastroenterology and Hepatology, 18(3):534–542, 2020.
  2. Predictive value of nlr, tils (cd4+/cd8+) and pd-l1 expression for prognosis and response to preoperative chemotherapy in gastric cancer. Cancer Immunology, Immunotherapy, 71(1):45–55, 2022.
  3. Treatment switch in poor responders with locally advanced gastric cancer after neoadjuvant chemotherapy. Annals of Surgical Oncology, 28(13):8892–8907, 2021.
  4. Phase ii study of neoadjuvant chemotherapy and extended surgery for locally advanced gastric cancer. Journal of British Surgery, 96(9):1015–1022, 2009.
  5. Response to neoadjuvant chemotherapy best predicts survival after curative resection of gastric cancer. Annals of Surgery, 229(3):303, 1999.
  6. Pathology and molecular biology of gastric cancer. Best Practice & Research Clinical Gastroenterology, 20(4):651–674, 2006.
  7. Rmdl: Recalibrated multi-instance deep learning for whole slide gastric image classification. Medical image analysis, 58:101549, 2019.
  8. Large-scale gastric cancer screening and localization using multi-task deep neural network. Neurocomputing, 448:290–300, 2021.
  9. 2d and 3d ct radiomic features performance comparison in characterization of gastric cancer: a multi-center study. IEEE journal of biomedical and health informatics, 25(3):755–763, 2020.
  10. Cluster-induced mask transformers for effective opportunistic gastric cancer screening on non-contrast ct scans. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 146–156. Springer, 2023.
  11. Deep learning predicts resistance to neoadjuvant chemotherapy for locally advanced gastric cancer: a multicenter study. Gastric Cancer, pages 1–10, 2022.
  12. Deep learning radiomics of ultrasonography can predict response to neoadjuvant chemotherapy in breast cancer at an early stage of treatment: a prospective study. European radiology, 32(3):2099–2109, 2022.
  13. A ct-based deep learning radiomics nomogram for predicting the response to neoadjuvant chemotherapy in patients with locally advanced gastric cancer: A multicenter cohort study. EClinicalMedicine, 46:101348, 2022.
  14. Single-shot pose estimation of surgical robot instruments’ shafts from monocular endoscopic images. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9960–9966. IEEE, 2020.
  15. A real-time state dependent region estimator for autonomous endoscope navigation. IEEE Transactions on Robotics, 37(3):918–934, 2020.
  16. St-mtl: Spatio-temporal multitask learning model to predict scanpath while tracking instruments in robotic surgery. Medical Image Analysis, 67:101837, 2021.
  17. Survival prediction of gastric cancer by a seven-microrna signature. Gut, 59(5):579–585, 2010.
  18. Prediction of overall survival and novel classification of patients with gastric cancer using the survival recurrent network. Annals of surgical oncology, 25:1153–1159, 2018.
  19. A deep learning risk prediction model for overall survival in patients with gastric cancer: A multicenter study. Radiotherapy and Oncology, 150:73–80, 2020.
  20. Deep learning radiomics nomogram based on enhanced ct to predict the response of metastatic lymph nodes to neoadjuvant chemotherapy in locally advanced gastric cancer. Annals of surgical oncology, pages 1–12, 2023.
  21. Multiparametric mri-based radiomics nomogram for early prediction of pathological response to neoadjuvant chemotherapy in locally advanced gastric cancer. European Radiology, 33(4):2746–2756, 2023.
  22. Estimation methods for time-dependent auc models with survival data. Canadian Journal of Statistics, 38(1):8–26, 2010.
  23. Summary measure of discrimination in survival models based on cumulative/dynamic time-dependent roc curves. Statistical methods in medical research, 25(5):2088–2102, 2016.
  24. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 6679–6687, 2021.
  25. Self-normalizing neural networks. Advances in neural information processing systems, 30, 2017.
  26. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  27. Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6546–6555, 2018.
  28. Attention-based deep multiple instance learning. In International conference on machine learning, pages 2127–2136. PMLR, 2018.
  29. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14318–14328, 2021.
  30. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in neural information processing systems, 34:2136–2147, 2021.
  31. Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18802–18812, 2022.
  32. Multiple instance learning framework with masked hard instance mining for whole slide image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4078–4087, 2023.
  33. Multi-modal multi-instance learning using weakly correlated histopathological images and tabular clinical information. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VIII 24, pages 529–539. Springer, 2021.
  34. Hfbsurv: hierarchical multimodal fusion with factorized bilinear models for cancer survival prediction. Bioinformatics, 38(9):2587–2594, 2022.
  35. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pages 6645–6649. Ieee, 2013.
  36. Memory fusion network for multi-view sequential learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  37. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2019, page 6558. NIH Public Access, 2019.
  38. Survival prediction of brain cancer with incomplete radiology, pathology, genomic, and demographic data. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 626–635. Springer, 2022.
  39. Com: Contrastive masked-attention model for incomplete multimodal learning. Neural Networks, 162:443–455, 2023.
  40. Rethinking attention with performers. arXiv preprint arXiv:2009.14794, 2020.
  41. Nyströmformer: A nyström-based algorithm for approximating self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14138–14148, 2021.
  42. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319–3328. PMLR, 2017.
  43. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
  44. A tumorigenic homeobox (hox) gene expressing human gastric cell line derived from putative gastric stem cell. European journal of gastroenterology & hepatology, 21(9):1016–1023, 2009.
  45. High level of homeobox a9 and pbx homeobox 3 expression in gastric cancer correlates with poor prognosis. Oncology letters, 14(5):5883–5889, 2017.
  46. Hoxa9 transcription factor is a double-edged sword: from development to cancer progression. Cancer and Metastasis Reviews, pages 1–20, 2023.
  47. A proteomic landscape of diffuse-type gastric cancer. Nature communications, 9(1):1012, 2018.
  48. Rbmx suppresses tumorigenicity and progression of bladder cancer by interacting with the hnrnp a1 protein to regulate pkm alternative splicing. Oncogene, 40(15):2635–2650, 2021.
  49. Construction of a survival nomogram for gastric cancer based on the cancer genome atlas of m6a-related genes. Frontiers in Genetics, 13:936658, 2022.
  50. Calmodulin 2 facilitates angiogenesis and metastasis of gastric cancer via stat3/hif-1a/vegf-a mediated macrophage polarization. Frontiers in Oncology, 11:727306, 2021.
  51. Knockdown of calm2 increases the sensitivity to afatinib in her2-amplified gastric cancer cells by regulating the akt/foxo3a/puma axis. Toxicology in Vitro, 87:105531, 2023.
  52. Nsun2 modified by sumo-2/3 promotes gastric cancer progression and regulates mrna m5c methylation. Cell Death & Disease, 12(9):842, 2021.
  53. Biowordvec, improving biomedical word embeddings with subword information and mesh. Scientific data, 6(1):52, 2019.
  54. Totalsegmentator: Robust segmentation of 104 anatomic structures in ct images. Radiology: Artificial Intelligence, 5(5), 2023.
  55. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  56. Transformer-based unsupervised contrastive learning for histopathological image classification. Medical image analysis, 81:102559, 2022.
  57. Gene2vec: distributed representation of genes based on co-expression. BMC genomics, 20:7–15, 2019.
  58. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
  59. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4025, 2021.
  60. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2016.
  61. Are multimodal transformers robust to missing modality? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18177–18186, 2022.
  62. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1103–1114, 2017.
  63. Efficient low-rank multimodal fusion with modality-specific factors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2018.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now