OphGLM: Training an Ophthalmology Large Language-and-Vision Assistant based on Instructions and Dialogue (2306.12174v2)
Abstract: Large multimodal LLMs (LMMs) have achieved significant success in general domains. However, due to the significant differences between medical images and text and general web content, the performance of LMMs in medical scenarios is limited. In ophthalmology, clinical diagnosis relies on multiple modalities of medical images, but unfortunately, multimodal ophthalmic LLMs have not been explored to date. In this paper, we study and construct an ophthalmic large multimodal model. Firstly, we use fundus images as an entry point to build a disease assessment and diagnosis pipeline to achieve common ophthalmic disease diagnosis and lesion segmentation. Then, we establish a new ophthalmic multimodal instruction-following and dialogue fine-tuning dataset based on disease-related knowledge data and publicly available real-world medical dialogue. We introduce visual ability into the LLM to complete the ophthalmic large language and vision assistant (OphGLM). Our experimental results demonstrate that the OphGLM model performs exceptionally well, and it has the potential to revolutionize clinical applications in ophthalmology. The dataset, code, and models will be made publicly available at https://github.com/ML-AILab/OphGLM.
- Vision–language model for visual question answering in medical imagery. Bioengineering, 10(3):380, 2023.
- Meddialog: a large-scale medical dialogue dataset. arXiv preprint arXiv:2004.03329, 2020.
- Pubmedclip: How much does clip benefit visual question answering in the medical domain? In Findings of the Association for Computational Linguistics: EACL 2023, pages 1151–1163, 2023.
- Medalpaca–an open-source collection of medical conversational ai models and training data. arXiv preprint arXiv:2304.08247, 2023.
- Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286, 2020.
- Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine. New England Journal of Medicine, 388(13):1233–1239, 2023. PMID: 36988602.
- The ai revolution in medicine: Gpt-4 and beyond, volume 2. The name of the publisher, 2023.
- Llava-med: Training a large language-and-vision assistant for biomedicine in one day. arXiv preprint arXiv:2306.00890, 2023.
- Self-supervised vision-language pretraining for medical visual question answering. arXiv preprint arXiv:2211.13594, 2022.
- Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening. Information Sciences, 501:511–522, 2019.
- Visual instruction tuning, 2023.
- Q2atransformer: Improving medical vqa via an answer querying decoder. In International Conference on Information Processing in Medical Imaging, pages 445–456. Springer, 2023.
- Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23(6), 2022.
- Capabilities of gpt-4 on medical challenge problems, 2023.
- OpenAI. Gpt-4 technical report, 2023.
- Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- Visual med-alpaca: A parameter-efficient biomedical llm with visual capabilities, 2023.
- Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
- Open-ended medical visual question answering through prefix tuning of language models. arXiv preprint arXiv:2303.05977, 2023.
- Biomedlm: a domain-specific large language model for biomedical text. MosaicML. Accessed: Dec, 23:3, 2022.
- Huatuo: Tuning llama model with chinese medical knowledge. arXiv preprint arXiv:2304.06975, 2023.
- Pmc-llama: Further finetuning llama on medical papers. arXiv preprint arXiv:2304.14454, 2023.
- Doctorglm: Fine-tuning your chinese doctor is not a herculean task. arXiv preprint arXiv:2304.01097, 2023.
- Chatdoctor: A medical chat model fine-tuned on llama model using medical domain knowledge. arXiv preprint arXiv:2303.14070, 2023.
- Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414, 2022.
- Large-scale domain-specific pretraining for biomedical vision-language processing. arXiv preprint arXiv:2303.00915, 2023.
- Weihao Gao (30 papers)
- Zhuo Deng (16 papers)
- Zhiyuan Niu (3 papers)
- Fuju Rong (2 papers)
- Chucheng Chen (2 papers)
- Zheng Gong (69 papers)
- Wenze Zhang (3 papers)
- Daimin Xiao (1 paper)
- Fang Li (142 papers)
- Zhenjie Cao (2 papers)
- Zhaoyi Ma (1 paper)
- Wenbin Wei (4 papers)
- Lan Ma (31 papers)