Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
143 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Data-Centric Foundation Models in Computational Healthcare: A Survey (2401.02458v2)

Published 4 Jan 2024 in cs.LG and cs.AI

Abstract: The advent of foundation models (FMs) as an emerging suite of AI techniques has struck a wave of opportunities in computational healthcare. The interactive nature of these models, guided by pre-training data and human instructions, has ignited a data-centric AI paradigm that emphasizes better data characterization, quality, and scale. In healthcare AI, obtaining and processing high-quality clinical data records has been a longstanding challenge, ranging from data quantity, annotation, patient privacy, and ethics. In this survey, we investigate a wide range of data-centric approaches in the FM era (from model pre-training to inference) towards improving the healthcare workflow. We discuss key perspectives in AI security, assessment, and alignment with human values. Finally, we offer a promising outlook of FM-based analytics to enhance the performance of patient outcome and clinical workflow in the evolving landscape of healthcare and medicine. We provide an up-to-date list of healthcare-related foundation models and datasets at https://github.com/Yunkun-Zhang/Data-Centric-FM-Healthcare .

Definition Search Book Streamline Icon: https://streamlinehq.com
References (333)
  1. Asma Ben Abacha and Pierre Zweigenbaum. 2011. Medical entity recognition: a comparaison of semantic and statistical methods. In Proceedings of BioNLP 2011 workshop. 56–64.
  2. Charu C Aggarwal and Philip S Yu. 2008. A general survey of privacy-preserving data mining models and algorithms. Privacy-preserving data mining: models and algorithms (2008), 11–52.
  3. Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis. NPJ digital medicine 4, 1 (2021), 65.
  4. Diffusion-based Data Augmentation for Skin Disease Classification: Impact Across Original Medical Datasets to Fully Synthetic Images. arXiv:2301.04802 [cs.LG]
  5. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems 35 (2022), 23716–23736.
  6. When is a Foundation Model a Foundation Model. arXiv:2309.11510 [cs.IR]
  7. Publicly available clinical BERT embeddings. arXiv preprint arXiv:1904.03323 (2019).
  8. Construction of the literature graph in semantic scholar. arXiv preprint arXiv:1805.02262 (2018).
  9. Abduladhim Ashtaiwi. 2022. Optimal histopathological magnification factors for deep learning-based breast cancer prediction. Applied System Innovation 5, 5 (2022), 87.
  10. A review on multimodal medical image fusion: Compendious analysis of medical modalities, multimodal databases, fusion techniques and quality metrics. Computers in biology and medicine 144 (2022), 105253.
  11. Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging. Nature Biomedical Engineering (2023), 1–24.
  12. Vlmo: Unified vision-language pre-training with mixture-of-modality-experts. Advances in Neural Information Processing Systems 35 (2022), 32897–32912.
  13. Big data analytics in healthcare. BioMed research international 2015 (2015).
  14. SciBERT: Pretrained Language Model for Scientific Text. In EMNLP. arXiv:arXiv:1903.10676
  15. The protein data bank. Nucleic acids research 28, 1 (2000), 235–242.
  16. Knowledge distillation: A good teacher is patient and consistent. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10925–10934.
  17. Integrative survival analysis of breast cancer with gene expression and DNA methylation data. Bioinformatics 37, 17 (2021), 2601–2608.
  18. Interpretable Medical Diagnostics with Structured Data Extraction by Large Language Models. arXiv preprint arXiv:2306.05052 (2023).
  19. Olivier Bodenreider. 2004. The unified medical language system (UMLS): integrating biomedical terminology. Nucleic acids research 32, suppl_1 (2004), D267–D270.
  20. Making the most of text semantics to improve biomedical vision–language processing. In European conference on computer vision. Springer, 1–21.
  21. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021).
  22. Deep orthogonal fusion: multimodal prognostic biomarker discovery integrating radiology, pathology, genomic, and clinical data. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part V 24. Springer, 667–677.
  23. Comparison of drug adherence rates among patients with seven different medical conditions. Pharmacotherapy: The Journal of Human Pharmacology and Drug Therapy 28, 4 (2008), 437–443.
  24. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
  25. Blood eosinophil levels as a biomarker in COPD. Respiratory medicine 138 (2018), 21–31.
  26. Fusion strategies for large-scale multi-modal image retrieval. Transactions on Large-Scale Data-and Knowledge-Centered Systems XXXIII (2017), 146–184.
  27. Carl A Burtis and David E Bruns. 2014. Tietz fundamentals of clinical chemistry and molecular diagnostics-e-book. Elsevier Health Sciences.
  28. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23). 5253–5270.
  29. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646 (2022).
  30. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19). 267–284.
  31. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21). 2633–2650.
  32. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision. 9650–9660.
  33. Adapting pretrained vision-language foundational models to medical imaging domains. arXiv preprint arXiv:2210.04133 (2022).
  34. Classification of Infant Sleep/Wake States: Cross-Attention among Large Scale Pretrained Transformer Networks using Audio, ECG, and IMU Data. arXiv preprint arXiv:2306.15808 (2023).
  35. Mining multi-center heterogeneous medical data with distributed synthetic learning. Nature communications 14, 1 (2023), 5510.
  36. A survey on evaluation of large language models. arXiv preprint arXiv:2307.03109 (2023).
  37. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16144–16155.
  38. Synthetic data in machine learning for medicine and healthcare. Nature Biomedical Engineering 5, 6 (2021), 493–497.
  39. A simple framework for contrastive learning of visual representations. In International conference on machine learning. PMLR, 1597–1607.
  40. Artificial intelligence for drug discovery: Resources, methods, and applications. Molecular Therapy-Nucleic Acids (2023).
  41. An Empirical Study of Training Self-Supervised Vision Transformers. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV). https://doi.org/10.1109/iccv48922.2021.00950
  42. Exploring the use of large language models for reference-free text quality evaluation: A preliminary empirical study. arXiv preprint arXiv:2304.00723 (2023).
  43. Empowering Psychotherapy with Large Language Models: Cognitive Distortion Detection through Diagnosis of Thought Prompting. arXiv preprint arXiv:2310.07146 (2023).
  44. Sam on medical images: A comprehensive study on three prompt modes. arXiv preprint arXiv:2305.00035 (2023).
  45. Accurate proteome-wide missense variant effect prediction with AlphaMissense. Science (2023), eadg7492.
  46. Cheng-Han Chiang and Hung-yi Lee. 2023. Can Large Language Models Be an Alternative to Human Evaluations? arXiv preprint arXiv:2305.01937 (2023).
  47. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022).
  48. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. Journal of clinical epidemiology 110 (2019), 12–22.
  49. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 (2022).
  50. Dobbs and the future of health data privacy for patients and healthcare organizations. Journal of the American Medical Informatics Association 30, 1 (2023), 155–160.
  51. 1000 Genomes Project Consortium et al. 2015. A global reference for human genetic variation. Nature 526, 7571 (2015), 68.
  52. ENCODE Project Consortium et al. 2012. An integrated encyclopedia of DNA elements in the human genome. Nature 489, 7414 (2012), 57.
  53. Deep learning for electroencephalogram (EEG) classification tasks: a review. Journal of neural engineering 16, 3 (2019), 031001.
  54. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501 (2018).
  55. scgpt: Towards building a foundation model for single-cell multi-omics using generative ai. bioRxiv (2023), 2023–04.
  56. Neuro-GPT: Developing A Foundation Model for EEG. arXiv preprint arXiv:2311.03764 (2023).
  57. Peter-Paul de Wolf. 2012. Statistical disclosure control. Wiley & Sons, Chichester.
  58. Segment anything model (sam) for digital pathology: Assess zero-shot segmentation on whole slide imaging. arXiv preprint arXiv:2304.04155 (2023).
  59. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
  60. Terrance DeVries and Graham W Taylor. 2017. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 (2017).
  61. Training a neural network based on unreliable human annotation of medical images. In 2018 IEEE 15th International symposium on biomedical imaging (ISBI 2018). IEEE, 39–42.
  62. Pathology-and-genomics multimodal transformer for survival outcome prediction. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 622–631.
  63. A large-scale synthetic pathological dataset for deep learning-enabled segmentation of breast cancer. Scientific Data 10, 1 (2023), 231.
  64. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence 5, 3 (2023), 220–235.
  65. A survey for in-context learning. arXiv preprint arXiv:2301.00234 (2022).
  66. Privacy for Free: How does Dataset Condensation Help Privacy? arXiv:2206.00240 [cs.CR]
  67. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020).
  68. Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunity. arXiv preprint arXiv:2305.08252 (2023).
  69. Understanding back-translation at scale. arXiv preprint arXiv:1808.09381 (2018).
  70. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5.
  71. Does clip benefit visual question answering in the medical domain as much as it does in the general domain? arXiv preprint arXiv:2112.13906 (2021).
  72. A guide to deep learning in healthcare. Nature medicine 25, 1 (2019), 24–29.
  73. Emilio Ferrara. 2023. Should chatgpt be biased? challenges and risks of bias in large language models. arXiv preprint arXiv:2304.03738 (2023).
  74. GET: a foundation model of transcription across human cell types. bioRxiv (2023), 2023–09.
  75. A gene prioritization method based on a swine multi-omics knowledgebase and a deep learning model. Communications Biology 3, 1 (2020), 502.
  76. Privacy-preserving data publishing: A survey of recent developments. ACM Computing Surveys (Csur) 42, 4 (2010), 1–53.
  77. Iason Gabriel. 2020. Artificial intelligence, values, and alignment. Minds and machines 30, 3 (2020), 411–437.
  78. Jevgenij Gamper and Nasir Rajpoot. 2021. Multiple instance captioning: Learning representations from histopathology textbooks and articles. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 16549–16559.
  79. Training Like a Medical Resident: Universal Medical Image Segmentation via Context Prior Learning. arXiv preprint arXiv:2306.02416 (2023).
  80. A data-scalable transformer for medical image segmentation: architecture, model efficiency, and benchmark. arXiv preprint arXiv:2203.00131 (2022).
  81. UTNet: a hybrid transformer architecture for medical image segmentation. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part III 24. Springer, 61–71.
  82. ChEMBL: a large-scale bioactivity database for drug discovery. Nucleic acids research 40, D1 (2012), D1100–D1107.
  83. Datasheets for datasets. Commun. ACM 64, 12 (2021), 86–92.
  84. Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text. Journal of Artificial Intelligence Research 77 (2023), 103–166.
  85. Parameter-Efficient Fine-Tuning of LLaMA for the Clinical Domain. arXiv preprint arXiv:2307.03042 (2023).
  86. Michael Gertz and Sushil Jajodia. 2007. Handbook of database security: applications and trends. Springer Science & Business Media.
  87. Large language model AI chatbots require approval as medical devices. Nature Medicine (2023), 1–3.
  88. How does ChatGPT perform on the medical licensing exams? The implications of large language models for medical education and knowledge assessment. MedRxiv (2022).
  89. Publishing data from electronic health records while preserving privacy: A survey of algorithms. Journal of biomedical informatics 50 (2014), 4–19.
  90. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH) 3, 1 (2021), 1–23.
  91. BiomedJourney: Counterfactual Biomedical Image Generation by Instruction-Learning from Multimodal Patient Journeys. arXiv preprint arXiv:2310.10765 (2023).
  92. Biosensors applications in medical field: A brief review. Sensors International 2 (2021), 100100.
  93. MedAlpaca–An Open-Source Collection of Medical Conversational AI Models and Training Data. arXiv preprint arXiv:2304.08247 (2023).
  94. Jigna J Hathaliya and Sudeep Tanwar. 2020. An exhaustive survey on security and privacy issues in Healthcare 4.0. Computer Communications 153 (2020), 311–335.
  95. Applying deep matching networks to Chinese medical question answering: A study and a dataset. BMC Medical Informatics and Decision Making 19, 2 (2019), 52. https://doi.org/10.1186/s12911-019-0761-8
  96. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 16000–16009.
  97. A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics. arXiv preprint arXiv:2310.05694 (2023).
  98. Accuracy of segment-anything model (sam) in medical image segmentation tasks. arXiv preprint arXiv:2304.09324 (2023).
  99. Wearable sensors: modalities, challenges, and prospects. Lab on a Chip 18, 2 (2018), 217–248.
  100. William Herring. 2019. Learning radiology: recognizing the basics. Elsevier Health Sciences.
  101. Blood biomarkers for mild traumatic brain injury: a selective review of unresolved issues. Biomarker research 9, 1 (2021), 1–17.
  102. William R Hogan and Michael M Wagner. 1997. Accuracy of data in computer-based patient records. Journal of the American Medical Informatics Association 4, 5 (1997), 342–355.
  103. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning. PMLR, 2790–2799.
  104. Chuanfei Hu and Xinde Li. 2023. When sam meets medical images: An investigation of segment anything model (sam) on multi-phase liver tumor segmentation. arXiv preprint arXiv:2304.08506 (2023).
  105. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021).
  106. How to Efficiently Adapt Large Segmentation Model (SAM) to Medical Images. arXiv preprint arXiv:2306.13731 (2023).
  107. SoK: Privacy-Preserving Data Synthesis. arXiv preprint arXiv:2307.02106 (2023).
  108. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ digital medicine 3, 1 (2020), 136.
  109. Segment anything model for medical images? arXiv preprint arXiv:2304.14660 (2023).
  110. A visual–language foundation model for pathology image analysis using medical Twitter. Nature Medicine (2023), 1–10.
  111. The role of artificial intelligence in early cancer diagnosis. Cancers 14, 6 (2022), 1524.
  112. Quilt-1M: One Million Image-Text Pairs for Histopathology. arXiv preprint arXiv:2306.11207 (2023).
  113. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33. 590–597.
  114. Bring Your Own Data! Self-Supervised Evaluation for Large Language Models. arXiv preprint arXiv:2306.13651 (2023).
  115. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning. PMLR, 4904–4916.
  116. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences 11, 14 (2021), 6421.
  117. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146 (2019).
  118. Mimic-iv. PhysioNet. Available online at: https://physionet. org/content/mimiciv/1.0/(accessed August 23, 2021) (2020).
  119. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data 6, 1 (2019), 317.
  120. MIMIC-III, a freely accessible critical care database. Scientific data 3, 1 (2016), 1–9.
  121. Justin M Johnson and Taghi M Khoshgoftaar. 2023. Data-Centric AI for Healthcare Fraud Detection. SN Computer Science 4, 4 (2023), 389.
  122. Integrating Al algorithms into the clinical workflow. Radiology: Artificial Intelligence 3, 6 (2021), e210013.
  123. Highly accurate protein structure prediction with AlphaFold. Nature 596, 7873 (2021), 583–589.
  124. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning. PMLR, 10697–10707.
  125. Benchmarking Self-Supervised Learning on Diverse Pathology Datasets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3344–3354.
  126. 100,000 histological images of human colorectal cancer and healthy tissue. https://doi.org/10.5281/zenodo.1214456
  127. Seven-point checklist and skin lesion classification using multitask multimodal neural nets. IEEE journal of biomedical and health informatics 23, 2 (2018), 538–546.
  128. Diffusion models in medical imaging: A comprehensive survey. Medical Image Analysis (2023), 102846.
  129. Developing a standardization algorithm for categorical laboratory tests for clinical big data research: retrospective study. JMIR medical informatics 7, 3 (2019), e14083.
  130. PubChem 2023 update. Nucleic acids research 51, D1 (2023), D1373–D1380.
  131. Segment anything. arXiv preprint arXiv:2304.02643 (2023).
  132. Large language models are zero-shot reasoners. Advances in neural information processing systems 35 (2022), 22199–22213.
  133. Michael R Kosorok and Eric B Laber. 2019. Annual review of statistics and its application. Precis Med 6 (2019), 263–286.
  134. Self-supervised learning in medicine and healthcare. Nature Biomedical Engineering 6, 12 (2022), 1346–1352.
  135. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25 (2012).
  136. Whole slide imaging (WSI) in pathology: current perspectives and future directions. Journal of digital imaging 33, 4 (2020), 1034–1040.
  137. A multi-organ nucleus segmentation challenge. IEEE transactions on medical imaging 39, 5 (2019), 1380–1391.
  138. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLoS digital health 2, 2 (2023), e0000198.
  139. Big healthcare data analytics: Challenges and applications. Handbook of large-scale distributed computing in smart healthcare (2017), 11–41.
  140. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36, 4 (2020), 1234–1240.
  141. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499 (2021).
  142. Does BERT pretrained on clinical notes reveal sensitive data? arXiv preprint arXiv:2104.07762 (2021).
  143. Eric Lehman and Alistair Johnson. 2023. Clinical-t5: Large language models built using mimic clinical text.
  144. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 (2021).
  145. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 (2019).
  146. Multimodal Foundation Models: From Specialists to General-Purpose Assistants. arXiv:2309.10020 [cs.CV]
  147. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. arXiv preprint arXiv:2306.00890 (2023).
  148. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 (2023).
  149. Huatuo-26M, a Large-scale Chinese Medical QA Dataset. arXiv:2305.01526 [cs.CL]
  150. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10965–10975.
  151. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 (2021).
  152. Polyp-sam: Transfer sam for polyp segmentation. arXiv preprint arXiv:2305.00293 (2023).
  153. Guo K Li HY. 2022. Blood Group Testing. Frontiers in medicine (2022).
  154. Advances, challenges and opportunities in creating data for trustworthy AI. Nature Machine Intelligence 4, 8 (2022), 669–677.
  155. Differentiate chatgpt-generated and human-written medical texts. arXiv preprint arXiv:2304.11567 (2023).
  156. Pmc-clip: Contrastive language-image pre-training using biomedical documents. arXiv preprint arXiv:2303.07240 (2023).
  157. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science 379, 6637 (2023), 1123–1130.
  158. Visual instruction tuning. arXiv preprint arXiv:2304.08485 (2023).
  159. DOLCE: A Model-Based Probabilistic Diffusion Framework for Limited-Angle CT Reconstruction. arXiv:2211.12340 [eess.IV]
  160. Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcare. arXiv preprint arXiv:2310.17956 (2023).
  161. Clip-driven universal model for organ segmentation and tumor detection. arXiv preprint arXiv:2301.00785 (2023).
  162. A text-guided protein design framework. arXiv preprint arXiv:2302.04611 (2023).
  163. Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models’ Alignment. arXiv:2308.05374 [cs.AI]
  164. Internchat: Solving vision-centric tasks by interacting with chatbots beyond language. arXiv preprint arXiv:2305.05662 (2023).
  165. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision. 10012–10022.
  166. Deid-gpt: Zero-shot medical text de-identification by gpt-4. arXiv preprint arXiv:2303.11032 (2023).
  167. S2ORC: The semantic scholar open research corpus. arXiv preprint arXiv:1911.02782 (2019).
  168. A comparative study on deep learning models for text classification of unstructured medical notes with various levels of class imbalance. BMC medical research methodology 22, 1 (2022), 181.
  169. MicroRNA expression profiles classify human cancers. nature 435, 7043 (2005), 834–838.
  170. Towards a Visual-Language Foundation Model for Computational Pathology. arXiv preprint arXiv:2307.12914 (2023).
  171. Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 19764–19775.
  172. BioGPT: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics 23, 6 (2022), bbac409.
  173. Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine. arXiv preprint arXiv:2308.09442 (2023).
  174. Translating radiology reports into plain language using chatgpt and gpt-4 with prompt learning: Promising results, limitations, and potential. arXiv preprint arXiv:2303.09038 (2023).
  175. Jun Ma and Bo Wang. 2023. Segment anything in medical images. arXiv preprint arXiv:2304.12306 (2023).
  176. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651 (2023).
  177. Wearable sensors for remote health monitoring. Sensors 17, 1 (2017), 130.
  178. Technical and policy approaches to balancing patient privacy and data sharing in clinical and translational research. Journal of Investigative Medicine 58, 1 (2010), 11–18.
  179. Segment anything model for medical image analysis: an experimental study. arXiv preprint arXiv:2304.10517 (2023).
  180. Bertalan Meskó and Eric J Topol. 2023. The imperative for regulatory oversight of large language models (or generative AI) in healthcare. npj Digital Medicine 6, 1 (2023), 120.
  181. Diana Mincu and Subhrajit Roy. 2022. Developing robust benchmarks for driving forward AI innovation in healthcare. Nature Machine Intelligence 4, 11 (2022), 916–921.
  182. Improving Zero-Shot Detection of Low Prevalence Chest Pathologies using Domain Pre-trained Language Models. arXiv:2306.08000 [physics.med-ph]
  183. Alok Mishra and Mukesh Verma. 2010. Cancer biomarkers: are we ready for the prime time? Cancers 2, 1 (2010), 190–208.
  184. A morphology focused diffusion probabilistic model for synthesis of histopathology images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2000–2009.
  185. Sunil Mohan and Donghui Li. 2019. Medmentions: A large biomedical corpus annotated with umls concepts. arXiv preprint arXiv:1902.09476 (2019).
  186. Foundation models for generalist medical artificial intelligence. Nature 616, 7956 (2023), 259–265.
  187. Med-Flamingo: a Multimodal Medical Few-shot Learner. arXiv preprint arXiv:2307.15189 (2023).
  188. Modeling community standards for metadata as templates makes data FAIR. Scientific Data 9, 1 (2022), 696.
  189. Varnakavi Naresh and Nohyun Lee. 2021. A review on biosensors and recent development of nanostructured materials-enabled biosensors. Sensors 21, 4 (2021), 1109.
  190. Privacy-preserving deep learning in medical informatics: applications, challenges, and solutions. Artificial Intelligence Review (2023), 1–43.
  191. The alignment problem from a deep learning perspective. arXiv preprint arXiv:2209.00626 (2022).
  192. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375 (2023).
  193. OpenAI. 2021. ChatGPT. https://openai.com/research/chatgpt.
  194. OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL]
  195. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023).
  196. Video-based AI for beat-to-beat assessment of cardiac function. Nature 580, 7802 (2020), 252–256.
  197. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (2022), 27730–27744.
  198. Six human-centered artificial intelligence grand challenges. International Journal of Human–Computer Interaction 39, 3 (2023), 391–437.
  199. Generation of Anonymous Chest Radiographs Using Latent Diffusion Models for Training Thoracic Abnormality Classification Systems. arXiv:2211.01323 [eess.IV]
  200. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on Health, Inference, and Learning. PMLR, 248–260.
  201. Unifying Large Language Models and Knowledge Graphs: A Roadmap. arXiv preprint arXiv:2306.08302 (2023).
  202. Privacy Risks of General-Purpose Language Models. In 2020 IEEE Symposium on Security and Privacy (SP). 1314–1331. https://doi.org/10.1109/SP40000.2020.00095
  203. Prediction of Alzheimer’s disease based on deep neural network by integrating gene expression and DNA methylation dataset. Expert Systems with Applications 140 (2020), 112873.
  204. Radiology Objects in COntext (ROCO): a multimodal image dataset. In Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis: 7th Joint International Workshop, CVII-STENT 2018 and Third International Workshop, LABELS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings 3. Springer, 180–189.
  205. A Study of Generative Large Language Model for Medical Research and Healthcare. arXiv preprint arXiv:2305.13523 (2023).
  206. Transfer learning in biomedical natural language processing: an evaluation of BERT and ELMo on ten benchmarking datasets. arXiv preprint arXiv:1906.05474 (2019).
  207. Brain imaging generation with latent diffusion models. In MICCAI Workshop on Deep Generative Models. Springer, 117–126.
  208. Large ai models in health informatics: Applications, challenges, and the future. arXiv preprint arXiv:2303.11568 (2023).
  209. Fusion of deep learning models of MRI scans, Mini–Mental State Examination, and logical memory test enhances diagnosis of mild cognitive impairment. Alzheimer’s & Dementia: Diagnosis, Assessment & Disease Monitoring 10 (2018), 737–749.
  210. Annotating 8,000 Abdominal CT Volumes for Multi-Organ Segmentation in Three Weeks. arXiv preprint arXiv:2305.09666 (2023).
  211. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748–8763.
  212. Improving language understanding by generative pre-training. (2018).
  213. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
  214. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21, 1 (2020), 5485–5551.
  215. AI in health and medicine. Nature medicine 28, 1 (2022), 31–38.
  216. Effect of scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations.
  217. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 (2022).
  218. Zero-shot text-to-image generation. In International Conference on Machine Learning. PMLR, 8821–8831.
  219. Assessing the utility of ChatGPT throughout the entire clinical workflow. medRxiv (2023), 2023–02.
  220. A survey of hallucination in large foundation models. arXiv preprint arXiv:2309.05922 (2023).
  221. Deep learning for medical image processing: Overview, challenges and the future. Classification in BioApps: Automation of Decision Making (2018), 323–350.
  222. Deep learning role in early diagnosis of prostate cancer. Technology in cancer research & treatment 17 (2018), 1533034618775530.
  223. Jerome P Reiter. 2012. Statistical approaches to protecting confidentiality for microdata and their effects on the quality of statistical inferences. Public opinion quarterly 76, 1 (2012), 163–181.
  224. Jerome P Reiter. 2019. Differential privacy and federal data releases. Annual review of statistics and its application 6 (2019), 85–101.
  225. Jerome P Reiter. 2023. Synthetic Data: A Look Back and A Look Forward. Trans. Data Priv. 16, 1 (2023), 15–24.
  226. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10684–10695.
  227. Sam. md: Zero-shot medical image segmentation capabilities of the segment anything model. arXiv preprint arXiv:2304.05396 (2023).
  228. Improving dermatology classifiers across populations using images generated by large diffusion models. arXiv:2211.13352 [eess.IV]
  229. ChatGPT applications in medical, dental, pharmacy, and public health education: A descriptive study highlighting the advantages and limitations. Narra J 3, 1 (2023), e103–e103.
  230. Appropriateness of cardiovascular disease prevention recommendations obtained from a popular online chat-based artificial intelligence model. JAMA 329, 10 (2023), 842–844.
  231. Local Differential Privacy for Artificial Intelligence of Medical Things. In Handbook of Security and Privacy of AI-Enabled Healthcare Systems and Internet of Medical Things. CRC Press, 241–270.
  232. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709 (2015).
  233. Decision tree and random forest models for outcome prediction in antibody incompatible kidney transplantation. Biomedical Signal Processing and Control 52 (2019), 456–462.
  234. Intern: A new learning paradigm towards general vision. arXiv preprint arXiv:2111.08687 (2021).
  235. Randstainna: Learning stain-agnostic features from histology slides by bridging stain augmentation and normalization. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 212–221.
  236. dbSNP: the NCBI database of genetic variation. Nucleic acids research 29, 1 (2001), 308–311.
  237. Detecting Pretraining Data from Large Language Models. arXiv preprint arXiv:2310.16789 (2023).
  238. Practical Medical Image Generation with Provable Privacy Protection based on Denoising Diffusion Probabilistic Models for High-resolution Volumetric Images. (2023).
  239. Connor Shorten and Taghi M Khoshgoftaar. 2019. A survey on image data augmentation for deep learning. Journal of big data 6, 1 (2019), 1–48.
  240. Text data augmentation for deep learning. Journal of big Data 8 (2021), 1–34.
  241. Visual Med-Alpaca: A Parameter-Efficient Biomedical LLM with Visual Capabilities. https://cambridgeltl.github.io/visual-med-alpaca/
  242. A review of analytics and clinical informatics in health care. Journal of medical systems 38 (2014), 1–7.
  243. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 15638–15650.
  244. Prerna Singh. 2023. Systematic review of data-centric approaches in artificial intelligence and machine learning. Data Science and Management (2023).
  245. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138 (2022).
  246. Towards expert-level medical question answering with large language models. arXiv preprint arXiv:2305.09617 (2023).
  247. Moco pretraining improves representation and transferability of chest x-ray models. In Medical Imaging with Deep Learning. PMLR, 728–744.
  248. A multi-modal convolutional neural network framework for the prediction of Alzheimer’s disease. In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, 1271–1274.
  249. Multimodal deep learning for biomedical data fusion: a review. Briefings in Bioinformatics 23, 2 (2022), bbab569.
  250. Mircea Steriade. 2005. Cellular substrates of brain rhythms. Electroencephalography: Basic principles, clinical applications and related fields (2005), 31–83.
  251. Multimodal fusion of imaging and genomics for lung cancer recurrence prediction. In 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). IEEE, 804–808.
  252. Pathasst: Redefining pathology through generative foundation ai assistant for pathology. arXiv preprint arXiv:2305.15072 (2023).
  253. Clinical intervention prediction and understanding using deep networks. arXiv preprint arXiv:1705.08498 (2017).
  254. UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches. Bioinformatics 31, 6 (2015), 926–932.
  255. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2818–2826.
  256. Terminology-aware medical dialogue generation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5.
  257. Does synthetic data generation of llms help clinical text mining? arXiv preprint arXiv:2303.04360 (2023).
  258. Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca.
  259. Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology. Medical image analysis 58 (2019), 101544.
  260. Understanding Unintended Memorization in Language Models Under Federated Learning. In Proceedings of the Third Workshop on Privacy in Natural Language Processing. Association for Computational Linguistics, Online, 1–10. https://doi.org/10.18653/v1/2021.privatenlp-1.1
  261. Large language models in medicine. Nature medicine (2023), 1–11.
  262. Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning. Nature Biomedical Engineering 6, 12 (2022), 1399–1406.
  263. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023).
  264. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023).
  265. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems 34 (2021), 200–212.
  266. Perturbation Methods for Protecting Data Privacy: A Review of Techniques and Applications. Automation and Machine Learning 4, 2 (2023), 31–41.
  267. Med-halt: Medical domain hallucination test for large language models. arXiv preprint arXiv:2307.15343 (2023).
  268. Opportunities and methodological challenges in EEG and MEG resting state functional brain network research. Clinical Neurophysiology 126, 8 (2015), 1468–1481.
  269. Richard Van Noorden and Jeffrey M Perkel. 2023. AI and science: what 1,600 researchers think. Nature 621, 7980 (2023), 672–675.
  270. Critical appraisal of artificial intelligence-based prediction models for cardiovascular disease. European heart journal 43, 31 (2022), 2921–2930.
  271. RadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Models. arXiv preprint arXiv:2305.01146 (2023).
  272. Attention is all you need. Advances in neural information processing systems 30 (2017).
  273. Clinical Text Summarization: Adapting Large Language Models Can Outperform Human Experts. arXiv:2309.07430 [cs.CL]
  274. Modern diagnosis of early esophageal cancer: from blood biomarkers to advanced endoscopy and artificial intelligence. Cancers 13, 13 (2021), 3162.
  275. Virchow: A Million-Slide Digital Pathology Foundation Model. arXiv preprint arXiv:2309.07778 (2023).
  276. SAMMedsuperscriptSAMMed\mathrm{SAM^{Med}}roman_SAM start_POSTSUPERSCRIPT roman_Med end_POSTSUPERSCRIPT: A medical image annotation framework based on large vision model. arXiv preprint arXiv:2307.05617 (2023).
  277. IvyGPT: InteractiVe Chinese pathwaY language model in medical domain. arXiv preprint arXiv:2307.10512 (2023).
  278. Shijun Wang and Ronald M Summers. 2012. Machine learning and radiology. Medical image analysis 16, 5 (2012), 933–951.
  279. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442 (2022).
  280. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. arXiv preprint arXiv:2305.11175 (2023).
  281. Chatgpt performs on the chinese national medical licensing examination. (2023).
  282. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2097–2106.
  283. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022).
  284. Transformer-based unsupervised contrastive learning for histopathological image classification. Medical image analysis 81 (2022), 102559.
  285. MediTab: Scaling Medical Tabular Data Predictors via Data Consolidation, Enrichment, and Refinement. arXiv:2305.12081 [cs.LG]
  286. Medclip: Contrastive learning from unpaired medical images and text. arXiv preprint arXiv:2210.10163 (2022).
  287. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.10904 (2021).
  288. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 (2021).
  289. Jason Wei and Kai Zou. 2019. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. arXiv preprint arXiv:1901.11196 (2019).
  290. The FAIR Guiding Principles for scientific data management and stewardship. Scientific data 3, 1 (2016), 1–9.
  291. Preparing medical imaging data for machine learning. Radiology 295, 1 (2020), 4–15.
  292. Leon Willenborg and Ton De Waal. 2012. Elements of statistical disclosure control. Vol. 155. Springer Science & Business Media.
  293. DrugBank 5.0: a major update to the DrugBank database for 2018. Nucleic acids research 46, D1 (2018), D1074–D1082.
  294. The Shaky Foundations of Clinical Foundation Models: A Survey of Large Language Models and Foundation Models for EMRs. arXiv preprint arXiv:2303.12961 (2023).
  295. Pmc-llama: Further finetuning llama on medical papers. arXiv preprint arXiv:2304.14454 (2023).
  296. Towards Generalist Foundation Model for Radiology. arXiv preprint arXiv:2308.02463 (2023).
  297. Medical sam adapter: Adapting segment anything model for medical image segmentation. arXiv preprint arXiv:2304.12620 (2023).
  298. NExT-GPT: Any-to-Any Multimodal LLM. arXiv preprint arXiv:2309.05519 (2023).
  299. Exploring the trade-offs: Unified large language models vs local fine-tuned models for highly-specific radiology nli task. arXiv preprint arXiv:2304.09138 (2023).
  300. Fei Xia and Meliha Yetisgen-Yildiz. 2012. Clinical corpus annotation: challenges and strategies. In Proceedings of the third workshop on building and evaluating resources for biomedical text mining (BioTxtM’2012) in conjunction with the international conference on language resources and evaluation (LREC), Istanbul, Turkey. 21–27.
  301. Group lasso regularized deep learning for cancer prognosis from multi-omics and clinical features. Genes 10, 3 (2019), 240.
  302. Unsupervised data augmentation for consistency training. Advances in neural information processing systems 33 (2020), 6256–6268.
  303. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9653–9663.
  304. DeepLesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning. Journal of medical imaging 5, 3 (2018), 036501–036501.
  305. Self-supervised electroencephalogram representation learning for automatic sleep staging: model development and evaluation study. JMIR AI 2, 1 (2023), e46769.
  306. MedMNIST v2-A large-scale lightweight benchmark for 2D and 3D biomedical image classification. Scientific Data 10, 1 (2023), 41.
  307. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601 (2023).
  308. LinkBERT: Pretraining Language Models with Document Links. In Association for Computational Linguistics (ACL).
  309. FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets. arXiv preprint arXiv:2307.10928 (2023).
  310. Towards General Purpose Medical AI: Continual Learning Medical Foundation Model. arXiv:2303.06580 [cs.CV]
  311. Shoulin Yin and Jing Bi. 2018. Medical image annotation based on deep transfer learning. In 2018 IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData). IEEE, 47–49.
  312. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 (2022).
  313. MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities. arXiv preprint arXiv:2308.02490 (2023).
  314. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432 (2021).
  315. Chatdoctor: A medical chat model fine-tuned on llama model using medical domain knowledge. arXiv preprint arXiv:2303.14070 (2023).
  316. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199 (2021).
  317. Data-centric artificial intelligence: A survey. arXiv preprint arXiv:2303.10158 (2023).
  318. Counterfactual memorization in neural language models. arXiv preprint arXiv:2112.12938 (2021).
  319. Cblue: A chinese biomedical language understanding evaluation benchmark. arXiv preprint arXiv:2106.08087 (2021).
  320. Shaoting Zhang and Dimitris Metaxas. 2023. On the Challenges and Perspectives of Foundation Models for Medical Image Analysis. arXiv preprint arXiv:2306.05705 (2023).
  321. Large-scale domain-specific pretraining for biomedical vision-language processing. arXiv preprint arXiv:2303.00915 (2023).
  322. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019).
  323. Knowledge-enhanced visual-language pre-training on chest radiology images. Nature Communications 14, 1 (2023), 4542.
  324. Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305.10415 (2023).
  325. Text-Guided Foundation Model Adaptation for Pathological Image Classification. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 272–282.
  326. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference. PMLR, 2–25.
  327. Yichi Zhang and Rushi Jiao. 2023. How Segment Anything Model (SAM) Boost Medical Image Segmentation? arXiv preprint arXiv:2305.03678 (2023).
  328. Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models. arXiv preprint arXiv:2309.01219 (2023).
  329. Input augmentation with sam: Boosting medical image segmentation with segmentation foundation model. arXiv preprint arXiv:2304.11332 (2023).
  330. Bo Zhao and Hakan Bilen. 2023. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 6514–6523.
  331. DeepOmix: A scalable and interpretable multi-omics deep learning framework and application in cancer survival analysis. Computational and structural biotechnology journal 19 (2021), 2719–2725.
  332. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. arXiv preprint arXiv:2302.09419 (2023).
  333. A foundation model for generalizable disease detection from retinal images. Nature (2023), 1–8.
Citations (17)

Summary

We haven't generated a summary for this paper yet.

Github Logo Streamline Icon: https://streamlinehq.com
X Twitter Logo Streamline Icon: https://streamlinehq.com