- The paper introduces OcularChat, a fine-tuned multimodal LLM that performs automated AMD diagnosis through image-grounded, interactive clinical dialogue.
- It leverages over 700K simulated physician-patient dialogues and 46K annotated CFPs with end-to-end adaptation, achieving superior diagnostic metrics.
- The evaluation demonstrates improved accuracy in advanced AMD detection and retinal feature classification compared to general-purpose models.
Multimodal Conversational AI for Retinal Image Interpretation in AMD: A Technical Analysis of OcularChat
Introduction
This work presents OcularChat, a fine-tuned multimodal LLM (MLLM) developed for the automated analysis and conversational diagnosis of age-related macular degeneration (AMD) from color fundus photographs (CFPs). OcularChat is specifically designed to move beyond static classification by providing image-grounded, interpretable outputs and engaging in clinically relevant dialogue, addressing the unique requirements of ophthalmic care where both diagnostic accuracy and succinct clinical explanation are critical. The system leverages a large dataset of simulated physician-patient dialogues, comprehensively annotated CFPs from the AREDS/AREDS2 studies, and full parameter end-to-end adaptation of Qwen2.5-VL.
The prevalence and clinical burden of AMD makes scalable and consistent interpretation of retinal fundus imagery an important target for AI-driven automation. While deep learning classifiers—such as DeepSeeNet and follow-on architectures—have achieved strong predictive performance on AMD grading tasks, these systems are limited in their ability to provide case-level clinical reasoning, respond interactively, or facilitate patient counseling demanded in real-world scenarios. The emergence of MLLMs, and specifically vision-LLMs (VLMs), suggests new opportunities for bridging the gap between image-based prediction and verifiable clinical communication. Current VLMs, however, are primarily trained on generic multi-domain data, exhibit limited exposure to ophthalmic language or fundus pathology, and thus lack reliability for domain-specialized tasks. Recent VLM benchmarks for ophthalmology highlight these gaps, showing that general-domain MLLMs are suboptimal for semantic reasoning and detailed retinal feature detection.
Methodology
Data Generation and Annotation
OcularChat is trained on 705,850 simulated dialogues paired with 46,167 CFPs, with corresponding demographic and clinical labels derived from the AREDS grading protocol. Dialogue generation was heavily physician-in-the-loop, using GPT-4V to synthesize both closed-format (structured Q/A for direct labels) and open-format (naturalistic, multi-turn conversation) dialogue. The clinical variables—late AMD status, drusen size, pigmentary abnormalities—are defined per AREDS. The dataset design ensures both high diagnostic annotation quality and realistic clinical question/answer patterns, which is critical for robust MLLM grounding.
Model Architecture and Training
OcularChat is initialized from Qwen2.5-VL-32B, an open-source, high-capacity VLM integrating a strong visual encoder and language backbone, optimized for multimodal conditioning. All model parameters (vision encoder, language decoder, MLP heads) are fine-tuned end-to-end with optimizer/parameter choices adhering to official Qwen2.5-VL protocols. Mixed-precision distributed training is performed on high-memory hardware, with extensive dialogue/image pairs driving cross-entropy minimization for conversational next-token prediction conditioned on both image embedding and dialogue context.
Evaluation Framework
Objective evaluation of OcularChat leverages accuracy and F1-score for three key tasks: advanced AMD detection, pigmentary abnormality identification, and drusen size categorization. The model’s structured outputs are compared to AREDS/AREDS2 reading center grades. Alongside conventional benchmarks (Qwen2.5-VL variants, LLaMA-Vision, MedGemma, GPT-01), external validation is performed on independent AREDS2 imagery. Furthermore, the study uses a custom ophthalmologist grading rubric for human evaluation—assessing not only diagnostic accuracy but also the coherence, appropriateness, and clinical value of generated conversational explanations on a 1-5 scale.
Results
OcularChat consistently exceeds the accuracy and F1 scores of both general-purpose and domain-adapted comparator VLMs across all major tasks. Explicitly:
- For advanced AMD detection on AREDS, OcularChat achieves 0.954 accuracy and 0.850 F1, outperforming GPT-01 (F1: 0.748) and MedGemma-4B (F1: 0.597).
- For pigmentary abnormalities, OcularChat records 0.849 accuracy and 0.803 F1, a marked advance over generalist Qwen2.5 baselines and other medical VLMs.
- For drusen size, typically the most challenging classification, it attains 0.678 accuracy and 0.642 F1, with the next-best models clustering around 0.4 F1.
External tests on AREDS2 confirm greater model robustness and transferability. OcularChat’s advanced AMD accuracy of 0.892 and drusen F1 of 0.495 present a much smaller performance drop compared to general VLMs.
Subjective Ophthalmologist Evaluation
OcularChat’s outputs receive substantially higher mean ratings than the reference (Qwen2.5-VL baseline) across all evaluated axes: advanced AMD (3.50 vs 2.83), pigmentary abnormality (3.27 vs 2.83), drusen (3.06 vs 2.43), and overall impression (2.98 vs 2.46) on the 5-point rubric. These evaluations show that fine-tuning with domain-specific dialogues and clinical supervision results not only in more accurate diagnostic statements but also in significantly improved clinical interpretability and user trust.
Inter-rater agreement (Cohen’s kappa) is higher for OcularChat, confirming the reproducibility and consistency of these improvements.
Model Behavior and Limitations
Qualitative reviews reveal that, although OcularChat explanations are concise and typically image-grounded, there remain instances of misalignment with clinical reasoning—e.g., emphasizing early-stage features in explanations for advanced AMD negatives rather than primary distinguishing signs (geographic atrophy, neovascular features). This highlights a remaining challenge in ensuring the explanatory narrative precisely tracks medically salient criteria for each task. In contrast to the verbose, sometimes hallucinated responses of the reference model, OcularChat’s pragmatic style is generally preferred by clinicians, who value clarity and directness in point-of-care settings.
The model is trained exclusively on CFPs, which—while broadly available—do not capture the full structural information spectrum attainable with modalities such as OCT. Thus, OcularChat’s deployment is best framed as augmentative in settings where CFP-based screening is the practical standard.
Practical and Theoretical Implications
OcularChat provides a robust demonstration of how domain-targeted MLLM fine-tuning can surpass generalist and lightly domain-adapted models in both objective and interpretability-driven performance metrics. The integration of realistic, context-aware, multi-turn dialogue and high-quality domain-specific supervision represents a practical path for medical AI adoption, particularly in fields where clinical communication and reasoning cannot be divorced from image-level prediction.
Furthermore, the model and code are released under open licenses, facilitating reproducibility and further domain specialization by the research community.
The primary advances substantiated are:
- MLLMs, when subjected to sufficient domain and conversation-focused supervision, can achieve high performance on both detection and dialogue generation tasks in medical imaging.
- Clinician-centric evaluation protocols surface semantic and reasoning misalignments undetectable in categorical accuracy assessment, a necessary step forward for AI safety in medicine.
- Extensible dialogue formats and human-in-the-loop supervision offer a foundation for future iterations involving richer imaging, heterogeneous clinical syntax (including multi-language support), and adaptive interaction styles based on environment or user preference.
Conclusion
OcularChat sets a technical benchmark for MLLMs in clinical image interpretation and conversational agent alignment in the context of AMD screening. The alignment between image-based outputs and interactive, clinically-oriented dialogue demonstrates how multimodal architectures can fulfill both diagnostic and communicative roles in high-stakes medical applications. While limitations persist in imaging modality coverage and nuanced stage-specific reasoning, the demonstrated methodology—comprising large-scale simulated dialogue supervision, full-parameter adaptation, and rigorous multi-axis evaluation—offers a robust blueprint for translation to other disease endpoints and specialties. Standardized, clinician-evaluated conversational assessment should be a required stage in the pipeline for all future medical MLLMs.