Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Towards Precision Healthcare: Robust Fusion of Time Series and Image Data (2405.15442v1)

Published 24 May 2024 in eess.IV, cs.CV, and cs.LG

Abstract: With the increasing availability of diverse data types, particularly images and time series data from medical experiments, there is a growing demand for techniques designed to combine various modalities of data effectively. Our motivation comes from the important areas of predicting mortality and phenotyping where using different modalities of data could significantly improve our ability to predict. To tackle this challenge, we introduce a new method that uses two separate encoders, one for each type of data, allowing the model to understand complex patterns in both visual and time-based information. Apart from the technical challenges, our goal is to make the predictive model more robust in noisy conditions and perform better than current methods. We also deal with imbalanced datasets and use an uncertainty loss function, yielding improved results while simultaneously providing a principled means of modeling uncertainty. Additionally, we include attention mechanisms to fuse different modalities, allowing the model to focus on what's important for each task. We tested our approach using the comprehensive multimodal MIMIC dataset, combining MIMIC-IV and MIMIC-CXR datasets. Our experiments show that our method is effective in improving multimodal deep learning for clinical applications. The code will be made available online.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042, 2019.
  2. Mimic-iv, a freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
  3. Medfuse: Multi-modal fusion with clinical time-series data and chest x-ray images. In Machine Learning for Healthcare Conference, pages 479–503. PMLR, 2022.
  4. Prediction of alzheimer’s progression based on multimodal deep-learning-based fusion and visual explainability of time-series data. Information Fusion, 92:363–388, 2023.
  5. Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction. Frontiers in Molecular Biosciences, 10:1136071, 2023.
  6. Integrated multimodal artificial intelligence framework for healthcare applications. NPJ digital medicine, 5(1):149, 2022.
  7. Mnn: multimodal attentional neural networks for diagnosis prediction. Extraction, 1(2019):A1, 2019.
  8. Mdf-net: Multimodal dual-fusion network for abnormality detection using cxr images and clinical data. arXiv preprint arXiv:2302.13390, 2023.
  9. Multimodal prompting with missing modalities for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14943–14952, 2023.
  10. M3care: Learning with missing modalities in multimodal healthcare data. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2418–2428, 2022.
  11. Multimodal learning with incomplete modalities by knowledge distillation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1828–1838, 2020.
  12. Are multimodal transformers robust to missing modality? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18177–18186, 2022.
  13. 3d deep learning for multi-modal imaging-guided survival time prediction of brain tumor patients. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19, pages 212–220. Springer, 2016.
  14. Multi-channel 3d deep feature learning for survival time prediction of brain tumor patients using multi-modal neuroimages. Scientific reports, 9(1):1103, 2019.
  15. B Srinivas and Gottapu Sasibhushana Rao. Segmentation of multi-modal mri brain tumor sub-regions using deep learning. Journal of Electrical Engineering & Technology, 15(4):1899–1909, 2020.
  16. Automated diagnosis of breast cancer using multi-modal datasets: A deep convolution neural network based approach. Biomedical Signal Processing and Control, 71:102825, 2022.
  17. Integrative data analysis of multi-platform cancer data with a multimodal deep learning approach. IEEE/ACM transactions on computational biology and bioinformatics, 12(4):928–937, 2014.
  18. A multimodal deep neural network for human breast cancer prognosis prediction by integrating multi-dimensional data. IEEE/ACM transactions on computational biology and bioinformatics, 16(3):841–850, 2018.
  19. Multimodal deep learning models for the prediction of pathologic response to neoadjuvant chemotherapy in breast cancer. Scientific reports, 11(1):18800, 2021.
  20. A multi-modal deep neural network for multi-class liver cancer diagnosis. Neural Networks, 165:553–561, 2023.
  21. Identifying breast cancer distant recurrences from electronic health records using machine learning. Journal of healthcare informatics research, 3:283–299, 2019.
  22. Deep learning for electronic health records analytics. IEEE Access, 7:101245–101259, 2019.
  23. Predicting glycaemia in type 1 diabetes patients: experiments in feature engineering and data imputation. Journal of healthcare informatics research, 4(1):71–90, 2020.
  24. A combined interpolation and weighted k-nearest neighbours approach for the imputation of longitudinal icu laboratory data. Journal of Healthcare Informatics Research, 4:174–188, 2020.
  25. A cross-sectional study to predict mortality for medicare patients based on the combined use of hcup tools. Journal of Healthcare Informatics Research, pages 1–19, 2021.
  26. Dilated recurrent neural networks for glucose forecasting in type 1 diabetes. Journal of Healthcare Informatics Research, 4:308–324, 2020.
  27. Bioberturk: Exploring turkish biomedical language model development strategies in low-resource setting. Journal of Healthcare Informatics Research, pages 1–14, 2023.
  28. Chest x-ray for predicting mortality and the need for ventilatory support in covid-19 patients presenting to the emergency department. European radiology, 31:1999–2012, 2021.
  29. Diabetes disease prediction using machine learning on big data of healthcare. In 2018 fourth international conference on computing communication control and automation (ICCUBEA), pages 1–6. IEEE, 2018.
  30. Machine learning for improved diagnosis and prognosis in healthcare. In 2017 IEEE aerospace conference, pages 1–9. IEEE, 2017.
  31. Lung cancer detection using image processing and machine learning healthcare. In 2018 International Conference on Current Trends towards Converging Technologies (ICCTCT), pages 1–5. IEEE, 2018.
  32. Vyshali J Gogi and MN Vijayalakshmi. Prognosis of liver disease: Using machine learning algorithms. In 2018 International Conference on Recent Innovations in Electrical, Electronics & Communication Engineering (ICRIEECE), pages 875–879. IEEE, 2018.
  33. Label dependent attention model for disease risk prediction using multimodal electronic health records. In 2021 IEEE International Conference on Data Mining (ICDM), pages 449–458. IEEE, 2021.
  34. Indication as prior knowledge for multimodal disease classification in chest radiographs with transformers. In 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pages 1–5. IEEE, 2022.
  35. M2-mixer: A multimodal mixer with multi-head loss for classification from multimodal data. In 2023 IEEE International Conference on Big Data (BigData), pages 1052–1058. IEEE, 2023.
  36. Predicting mortality among patients with liver cirrhosis in electronic health records with machine learning. PloS one, 16(8):e0256428, 2021.
  37. Multimodal learning for predicting mortality in patients with pulmonary arterial hypertension. In 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 2704–2710. IEEE, 2022.
  38. Contrast limited adaptive histogram equalization based enhancement for real time video system. In 2014 International Conference on Advances in Computing, Communications and Informatics (ICACCI), pages 2392–2397, 2014.
  39. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  40. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  41. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997.
  42. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
  43. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7482–7491, 2018.
  44. A simple framework for contrastive learning of visual representations, 2020.
  45. Multimodal contrastive training for visual representation learning, 2021.
  46. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
  47. Card: Classification and regression diffusion models. Advances in Neural Information Processing Systems, 35:18100–18115, 2022.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets