Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

HyperFusion: A Hypernetwork Approach to Multimodal Integration of Tabular and Medical Imaging Data for Predictive Modeling (2403.13319v1)

Published 20 Mar 2024 in cs.CV, cs.LG, and eess.IV

Abstract: The integration of diverse clinical modalities such as medical imaging and the tabular data obtained by the patients' Electronic Health Records (EHRs) is a crucial aspect of modern healthcare. The integrative analysis of multiple sources can provide a comprehensive understanding of a patient's condition and can enhance diagnoses and treatment decisions. Deep Neural Networks (DNNs) consistently showcase outstanding performance in a wide range of multimodal tasks in the medical domain. However, the complex endeavor of effectively merging medical imaging with clinical, demographic and genetic information represented as numerical tabular data remains a highly active and ongoing research pursuit. We present a novel framework based on hypernetworks to fuse clinical imaging and tabular data by conditioning the image processing on the EHR's values and measurements. This approach aims to leverage the complementary information present in these modalities to enhance the accuracy of various medical applications. We demonstrate the strength and the generality of our method on two different brain Magnetic Resonance Imaging (MRI) analysis tasks, namely, brain age prediction conditioned by subject's sex, and multiclass Alzheimer's Disease (AD) classification conditioned by tabular data. We show that our framework outperforms both single-modality models and state-of-the-art MRI-tabular data fusion methods. The code, enclosed to this manuscript will be made publicly available.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (78)
  1. Hypernetwork-based adaptive image restoration, in: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE. pp. 1–5.
  2. An open resource for transdiagnostic research in pediatric mental health and learning disorders. Scientific data 4, 1–26.
  3. Bottom-up and top-down attention for image captioning and visual question answering, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6077–6086.
  4. An efficient 3D deep convolutional network for Alzheimer’s disease diagnosis using MR images, in: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 149–153.
  5. Toward discovery science of human brain function. Proceedings of the national academy of sciences 107, 4734–4739.
  6. Deep neural networks and tabular data: a survey. IEEE Transactions on Neural Networks and Learning Systems .
  7. Brain Genomics Superstruct Project (GSP).
  8. Longitudinal multiple sclerosis lesion segmentation: resource and challenge. NeuroImage 148, 77–102.
  9. Principled weight initialization for hypernetworks, in: International Conference on Learning Representations.
  10. A hybrid machine learning/deep learning COVID-19 severity predictive model from CT images and clinical data. Scientific reports 12, 4329.
  11. Sex differences in brain aging: a quantitative magnetic resonance imaging study. Archives of neurology 55, 169–179.
  12. Predicting age using neuroimaging: innovative brain ageing biomarkers. Trends in neurosciences 40, 681–690.
  13. LIMITR: Leveraging local information for medical image-text representation, in: International Conference on computer Vision.
  14. The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism. Molecular psychiatry 19, 659–667.
  15. A deep generative multimodal imaging genomics framework for alzheimer’s disease prediction, in: 2022 IEEE 22nd International Conference on Bioinformatics and Bioengineering (BIBE), pp. 41–44.
  16. Multimodal multitask deep learning model for Alzheimer’s disease progression detection based on time series data. Neurocomputing 412, 197–215.
  17. The australian imaging, biomarkers and lifestyle (aibl) study of aging: methodology and baseline characteristics of 1112 individuals recruited for a longitudinal study of alzheimer’s disease. International psychogeriatrics 21, 672–687.
  18. End-to-end Alzheimer’s disease diagnosis and biomarker identification, in: Shi, Y., Suk, H.I., Liu, M. (Eds.), Machine Learning in Medical Imaging, Springer International Publishing, Cham. pp. 337–345.
  19. Estimating brain age based on a uniform healthy population with deep learning and structural magnetic resonance imaging. Neurobiology of aging 91, 15–25.
  20. Ten years of BrainAGE as a neuroimaging biomarker of brain aging: what insights have we gained? Frontiers in neurology , 789.
  21. Prevalence of Alzheimer’s disease and other dementias in an elderly urban population. Neurology 41, 1886–1886.
  22. Diagnostic and sex effects on limbic volumes in early-onset bipolar disorder and schizophrenia. Schizophrenia bulletin 34, 37–46.
  23. Understanding the difficulty of training deep feedforward neural networks, in: Proceedings of the thirteenth international conference on artificial intelligence and statistics, JMLR Workshop and Conference Proceedings. pp. 249–256.
  24. Nipype: a flexible, lightweight and extensible neuroimaging data processing framework in python. Frontiers in neuroinformatics 5, 13.
  25. Hypernetworks. CoRR abs/1609.09106 (2016). arXiv preprint arXiv:1609.09106 .
  26. Best of both worlds: Multimodal contrastive learning with tabular and imaging data, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 23924–23935.
  27. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, in: Proceedings of the IEEE international conference on computer vision, pp. 1026–1034.
  28. Identity mappings in deep residual networks, in: Leibe, B., Matas, J., Sebe, N., Welling, M. (Eds.), Computer Vision – ECCV 2016, Springer International Publishing, Cham. pp. 630–645.
  29. Information extraction from medical images: developing an e-science application based on the globus toolkit, in: Proceedings of the 2nd UK e-Science All Hands Meeting.
  30. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ digital medicine 3, 136.
  31. Robust brain extraction across datasets and comparison with publicly available methods. IEEE Transactions on Medical Imaging 30, 1617–1634.
  32. The alzheimer’s disease neuroimaging initiative (adni): Mri methods. Journal of Magnetic Resonance Imaging: An Official Journal of the International Society for Magnetic Resonance in Medicine 27, 685–691.
  33. Fsl. NeuroImage 62, 782–790. 20 YEARS OF fMRI.
  34. Deep learning-based brain age prediction in normal aging and dementia. Nature Aging 2, 412–424.
  35. Are sex and educational level independent predictors of dementia and alzheimer’s disease? incidence data from the paquid project. Journal of Neurology, Neurosurgery & Psychiatry 66, 177–183.
  36. From a deep learning model back to the brain—identifying regional predictors and their relation to aging. Human Brain Mapping 41, 3235–3252.
  37. Deep meta functionals for shape representation, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1824–1833.
  38. Joint classification and regression via deep multi-task multi-channel learning for Alzheimer’s disease diagnosis. IEEE Transactions on Biomedical Engineering 66, 1195–1206.
  39. Longitudinal test-retest neuroimaging data from healthy young adults in southwest china. Scientific data 4, 1–9.
  40. Open access series of imaging studies: longitudinal mri data in nondemented and demented older adults. Journal of cognitive neuroscience 22, 2677–2684.
  41. Open access series of imaging studies (oasis): cross-sectional mri data in young, middle aged, nondemented, and demented older adults. Journal of cognitive neuroscience 19, 1498–1507.
  42. The parkinson progression marker initiative (ppmi). Progress in Neurobiology 95, 629–635. Biological Markers for Neurodegenerative Diseases.
  43. Functional imaging of the hemodynamic sensory gating response in schizophrenia. Human brain mapping 34, 2302–2312.
  44. A probabilistic atlas of the human brain: Theory and rationale for its development: The international consortium for brain mapping (icbm). NeuroImage 2, 89–101.
  45. The multimodal brain tumor image segmentation benchmark (brats). IEEE transactions on medical imaging 34, 1993–2024.
  46. The nki-rockland sample: a model for accelerating the pace of discovery science in psychiatry. Frontiers in neuroscience 6, 152.
  47. Deep learning-based late fusion of multimodal information for emotion classification of music video. Multimedia Tools and Applications 80, 2887–2905.
  48. Accurate brain age prediction with lightweight deep neural networks. Medical Image Analysis 68, 101871.
  49. FiLM: visual reasoning with a general conditioning layer. Proceedings of the AAAI Conference on Artificial Intelligence 32.
  50. Analysing race and sex bias in brain age prediction, in: Workshop on Clinical Image-Based Procedures, Springer. pp. 194–204.
  51. Genetic variants of foxp2 and kiaa0319/ttrap/them2 locus are associated with altered brain activation in distinct language-related regions. Journal of Neuroscience 32, 817–825.
  52. A phenome-wide examination of neural and cognitive function. Scientific data 3, 1–12.
  53. Multi-modal deep learning models for Alzheimer’s disease prediction using MRI and EHR, in: 2022 IEEE 22nd International Conference on Bioinformatics and Bioengineering (BIBE), IEEE. pp. 168–173.
  54. Fusion of deep learning models of MRI scans, mini–mental state examination, and logical memory test enhances diagnosis of mild cognitive impairment. Alzheimer’s and Dementia: Diagnosis, Assessment and Disease Monitoring 10, 737–749.
  55. Learning transferable visual models from natural language supervision, in: International conference on machine learning, PMLR. pp. 8748–8763.
  56. Hippocampus and its involvement in Alzheimer’s disease: a review. 3 Biotech 12.
  57. Evaluating the impact of intensity normalization on mr image synthesis. CoRR abs/1812.04652. arXiv:1812.04652.
  58. Adult hippocampal neurogenesis in Alzheimer’s disease: A roadmap to clinical relevance. Cell Stem Cell 30, 120--136.
  59. The cambridge centre for ageing and neuroscience (cam-can) study protocol: a cross-sectional, lifespan, multidisciplinary examination of healthy cognitive ageing. BMC neurology 14, 1--25.
  60. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 .
  61. A multi-modal convolutional neural network framework for the prediction of Alzheimer’s disease, in: 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 1271--1274.
  62. Uk biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS medicine 12, e1001779.
  63. Data-driven multimodal fusion: approaches and applications in psychiatric research. Psychoradiology , kkad026.
  64. N4itk: Improved n3 bias correction. IEEE Transactions on Medical Imaging 29, 1310--1320.
  65. Multimodal deep learning models for early detection of Alzheimer’s disease stage. Scientific reports 11, 3254.
  66. Show and tell: A neural image caption generator, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3156--3164.
  67. Learning deep structure-preserving image-text embeddings, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  68. Classification of Alzheimer’s disease based on eight-layer convolutional neural network with leaky rectified linear unit and max pooling. Journal of medical systems 42, 1--11.
  69. Medclip: Contrastive learning from unpaired medical images and text. arXiv preprint arXiv:2210.10163 .
  70. Convolutional neural networks for classification of Alzheimer’s disease: Overview and reproducible evaluation. Medical image analysis 63, 101694.
  71. DAFT: A universal module to interweave tabular data and 3D images in CNNs. NeuroImage 260, 119505.
  72. HyperTab: Hypernetwork approach for deep learning on small tabular datasets. arXiv preprint arXiv:2304.03543 .
  73. Show, attend and tell: Neural image caption generation with visual attention, in: International conference on machine learning, PMLR. pp. 2048--2057.
  74. Flexible fusion network for multi-modal brain tumor segmentation. IEEE Journal of Biomedical and Health Informatics 27, 3349--3359.
  75. Text-to-image diffusion model in generative ai: A survey. arXiv preprint arXiv:2303.07909 .
  76. Stackgan++: Realistic image synthesis with stacked generative adversarial networks. IEEE transactions on pattern analysis and machine intelligence 41, 1947--1962.
  77. Effective feature learning and fusion of multimodality data using stage-wise deep neural network for dementia diagnosis. Human brain mapping 40, 1001--1016.
  78. An open science resource for establishing reliability and reproducibility in functional connectomics. Scientific data 1, 1--13.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Daniel Duenias (1 paper)
  2. Brennan Nichyporuk (17 papers)
  3. Tal Arbel (41 papers)
  4. Tammy Riklin Raviv (9 papers)

Summary

HyperFusion: A Hypernetwork Approach to Multimodal Integration of Tabular and Medical Imaging Data for Predictive Modeling

The integration of multimodal data in medical applications plays a pivotal role in enhancing the predictive capabilities of automated systems. The paper "HyperFusion: A Hypernetwork Approach to Multimodal Integration of Tabular and Medical Imaging Data for Predictive Modeling" presents a novel framework designed to integrate tabular data, commonly derived from Electronic Health Records (EHRs), with medical imaging data through a hypernetwork architecture. This approach demonstrates significant potential in improving predictive outcomes in tasks such as brain age prediction and Alzheimer's Disease (AD) classification.

Methodology

The researchers introduce a deep learning framework leveraging hypernetworks, a secondary network that generates adaptive parameters conditionally for a primary neural network. The hypernetwork in "HyperFusion" processes the tabular data to influence the image analysis network's weights, thereby conditioning the imaging network's task output based on EHR values.

The core methodology involves:

  • Hypernetwork and Primary Network Composition: The hypernetwork generates task-specific weights and biases, contributing to a dynamic adaptation of image processing conditioned by tabular attributes. The integration allows for nuanced data handling and tailored parameter adjustments during the fusion process.
  • Embedding Tabular Data: Using an MLP-based embedding architecture, the framework embeds the tabular data into latent features, enabling complex interactions and pattern recognition within tabular modalities.
  • Loss Functions and Regularization: The framework employs a loss that combines task-specific outcomes with regularization, optimizing both the primary and hypernetwork parameters through backpropagation.

Experimental Validation

The paper validates the "HyperFusion" framework using two primary medical tasks:

  1. Brain Age Prediction: The framework conditioned brain MRI analysis on the subject’s sex, improving prediction accuracy compared to uninformed models. The results underscore the impact of incorporating demographic data into predictive modeling.
  2. Alzheimer's Disease Classification: By utilizing data from the ADNI database, the framework effectively classifies CN, MCI, and AD categories using cropped MRI data (targeting the hippocampus) integrated with selected tabular attributes like demographics and biomarkers. The paper finds enhanced classification metrics over competing fusion methodologies, highlighting the potential of hypernetworks in complex diagnostic tasks.

Results and Implications

The "HyperFusion" framework shows a marked improvement over baseline models that rely on single modalities. The results indicate statistically significant advances in prediction accuracy and model robustness across both application scenarios. For brain age prediction, MAE scores were improved when using the hypernetwork, demonstrating that sex-based conditioning enhances predictive accuracy. Similarly, in AD classification, the framework outperformed state-of-the-art models in balanced accuracy and macro-AUC scores, ensuring a holistic integration of multimodal inputs.

Future Directions

HyperFusion's comprehensive framework serves as a precursor for future research into more complex data integration challenges in AI. Potential developments include:

  • Reverse Conditioning: Exploring the inverse conditioning process where image data refines EHR analysis.
  • Alternative Architectures: Adapting this hypernetwork approach to architectures like Transformers could broaden applicability across diverse domains beyond current imaging and tabular data correlations.

In conclusion, the HyperFusion framework marks a significant stride forward in multimodal data integration within predictive modeling in healthcare. The adaptability and enhanced performance offer a valuable addition to the repository of tools available to AI researchers and clinicians striving for precise and comprehensive diagnostic systems.