MM-Retinal-Reason: Ophthalmic Multimodal Reasoning

• MM-Retinal-Reason is a comprehensive ophthalmic dataset integrating basic visual discrimination and complex clinical reasoning across varied imaging modalities.
• It organizes four interlinked subsets—from basic VQA to complex multi-image cases—to benchmark both perceptual and clinical inference tasks.
• Paired with OphthaReason and enhanced via Uncertainty-Aware Dynamic Thinking, the framework achieves notable performance gains over existing models.

MM-Retinal-Reason is an ophthalmic multimodal dataset designed to cover the full spectrum from basic visual perception to complex clinical reasoning, and it is paired with OphthaReason, an ophthalmology-specific multimodal reasoning model with step-by-step reasoning traces. The resource is organized around four interlinked subsets—basic reasoning VQA, chain-of-thought reasoning trajectories, image–caption pairs, and fully multimodal multi-image clinical cases—with the stated aim of enhancing visual-centric fundamental reasoning capabilities and emulating realistic clinical thinking patterns in retinal AI (Wu et al., 22 Aug 2025).

1. Conceptual scope

MM-Retinal-Reason defines ophthalmic reasoning as a spectrum rather than a single task class. In its taxonomy, basic reasoning is explicitly defined as “shallow inference from visual cues,” including modality recognition, anatomic orientation, and simple lesion identification. Complex reasoning tasks require integrating multiple images such as CFP, FFA, and OCT with heterogeneous text such as chief complaint, history of present illness, and physical examination to derive a differential and final diagnosis (Wu et al., 22 Aug 2025).

A common misconception is to treat ophthalmic multimodal reasoning as equivalent to image-only VQA or generic lesion classification. MM-Retinal-Reason explicitly separates these regimes. Its basic subset covers True/False, single-answer multiple-choice, and multi-answer multiple-choice formats, while its complex subset is case-based and embeds presentation text, question, explicit chain-of-thought, image captions, and ground-truth answer within the same sample structure (Wu et al., 22 Aug 2025).

This separation is significant because it formalizes two different inference burdens. The first concerns visual discrimination from retinal imagery; the second concerns clinical integration across modalities and text sources. A plausible implication is that the benchmark was designed not only to test perceptual competence but also to probe whether multimodal models can approximate realistic ophthalmic diagnostic workflows.

2. Dataset composition

MM-Retinal-Reason comprises four linked subsets with distinct supervision granularity and reasoning demands (Wu et al., 22 Aug 2025).

Subset	Content	Size
Basic Reasoning VQA	CFP, FFA, OCT image–question pairs	CFP: 55,224; FFA: 34,159; OCT: 131,985
CoT Reasoning Trajectories	Step-by-step traces for sampled basic VQA items	CFP: 10,270; FFA: 3,274; OCT: 2,343
Image–Caption Pairs	Multimodal ophthalmic image–caption pairs	80,520
Complex Reasoning VQA	Fully multimodal, multi-image clinical cases	5,027 cases; 400 held out for test

The basic VQA subset covers more than 100 retinal diseases and lesions, including diabetic retinopathy, AMD, and glaucoma. The image–caption subset spans CFP, FFA, OCT, CT, slit lamp, ultrasound, and related ophthalmic imagery. The complex subset is organized at the case level rather than the image level, which is crucial because each case includes a “Presentation” block derived from the source paper, alongside image captions, question, explicit reasoning trace, and answer (Wu et al., 22 Aug 2025).

The image–caption component was extracted from ophthalmology articles in PubMed Central through figure decomposing with Dab-DETR and subcaption parsing with Qwen2.5-VL-72B. This indicates that MM-Retinal-Reason is not restricted to fundus photography alone, even though retinal modalities remain central to its reasoning tasks (Wu et al., 22 Aug 2025).

3. Annotation workflow and reasoning supervision

The dataset construction pipeline combines automated generation, model-assisted structuring, and specialist review. For the chain-of-thought subset, the procedure is specified as a three-stage sequence: Qwen2.5-VL-72B first produces a concise image description, DeepSeek-R1 then generates a step-by-step reasoning trace ending in the correct answer, and an MLLM rubric validates the result along four dimensions: image description clarity, reasoning transparency, validity and integrity of logical steps, and clinical rigor (Wu et al., 22 Aug 2025).

The complex reasoning subset follows a separate four-step pipeline: rule-based filtering of PMC case reports, structured QA generation with o4-mini, automated quality evaluation using eight metrics under case completeness, reasoning depth, and clinical significance, and visual information alignment with hallucination checking. Each retained case therefore encodes not only multimodal evidence but also an explicit quality-controlled reasoning scaffold (Wu et al., 22 Aug 2025).

Human oversight is central to the curation process. All CoT traces and complex-case extracts were reviewed by a panel of ophthalmologists with 5+ years clinical experience. Automated MLLM-based scoring was used to ensure consistency, low-scoring samples were discarded, and the remaining samples passed a secondary expert audit. At the same time, the report states that no single numeric inter-annotator-agreement statistic such as kappa is reported. That absence is an important methodological qualifier when comparing MM-Retinal-Reason with datasets that foreground formal agreement estimates (Wu et al., 22 Aug 2025).

4. OphthaReason architecture and uncertainty-aware adaptation

OphthaReason is built on two open-source MLLM backbones, yielding Qwen-based and Intern-based variants. The visual encoder is a fine-tuned vision encoder from either Qwen2.5-VL-3B visions or InternVL3-2B’s vision tower, while the language backbone is Qwen2.5-VL-3B-Instruct or InternVL3-2B. Multimodal fusion uses the standard cross-attention mechanism of encoder–decoder transformers, with decoder-layer updates proceeding through self-attention over prior tokens, cross-attention against image embeddings, and feed-forward transformation (Wu et al., 22 Aug 2025).

The model’s distinctive component is Uncertainty-Aware Dynamic Thinking (UADT), which shapes RL updates according to sample-level uncertainty. Token-level entropy and sequence-level entropy are defined as

$$H_t = -\sum_{v\in V} \pi(y_t=v|\cdot)\log \pi(y_t=v|\cdot), \qquad H_{\mathrm{seq}} = \frac{1}{T}\sum_{t=1}^{T} H_t.$$

The adaptive entropy threshold follows an EMA update,

$$\tau_s = \alpha\cdot\tau_{s-1} + (1-\alpha)\cdot(\text{average }H_{\mathrm{seq}}\text{ in batch}),$$

with $\tau_0$ initialized as the median entropy on base-model inference. UADT then defines an uncertainty factor

$$\omega = \tanh[\gamma\cdot(H_{\mathrm{seq},i}-\tau_s)], \qquad \gamma=0.5,$$

and augments the GRPO-style advantage

$A_i = \frac{r_i-\mathrm{mean}(R)}{\mathrm{std}(R)}$

into

$$A_i^{\mathrm{UADT}} = A_i + \lambda\cdot\omega_{i,s}\cdot H_{\mathrm{seq},i}, \qquad \lambda=1.0.$$

The final RL objective is formulated as a clipped group relative policy optimization objective with KL regularization (Wu et al., 22 Aug 2025).

An important clarification follows directly from the specification: UADT does not mean that the model sometimes emits short reasoning traces and sometimes long ones. The model always emits a full CoT trace; UADT instead modulates exploration depth by assigning larger policy updates to high-entropy samples and smaller updates to low-entropy samples. This distinction matters because it separates reasoning-format output from optimization-time allocation of learning effort (Wu et al., 22 Aug 2025).

5. Training pipeline and optimization regime

OphthaReason is trained in three stages. Stage 1, Vision–Language Alignment, uses 72,861 MM-Retinal-Reason image–caption pairs together with 9,437 samples from MM-Retinal. The LLM is frozen, and only the vision encoder and cross-modal projector are fine-tuned for 1 epoch; the reported learning rates are $2\times10^{-6}$ for the Qwen variant and $2\times10^{-5}$ for the Intern variant (Wu et al., 22 Aug 2025).

Stage 2, CoT Supervised Fine-Tuning, uses 20,308 CoT samples and 28,238 VQA samples from the basic and complex subsets. The full model is fine-tuned for 2 epochs with standard cross-entropy over the concatenated CoT tokens and answer, using learning rates of $1\times10^{-6}$ for Qwen and $1\times10^{-5}$ for Intern (Wu et al., 22 Aug 2025).

Stage 3, Reinforcement Learning with UADT, reuses the same training split as Stage 2 for 2 epochs at learning rate $$\tau_s = \alpha\cdot\tau_{s-1} + (1-\alpha)\cdot(\text{average }H_{\mathrm{seq}}\text{ in batch}),$$0, with AdamW and temperature $$\tau_s = \alpha\cdot\tau_{s-1} + (1-\alpha)\cdot(\text{average }H_{\mathrm{seq}}\text{ in batch}),$$1. The reward is the sum of FormatReward and AccuracyReward. FormatReward checks tag conformity—> …<answer>…</answer>, and additionally <caption>…</caption> for open-ended settings. AccuracyReward uses IoU for multiple-choice outputs,

$$\tau_s = \alpha\cdot\tau_{s-1} + (1-\alpha)\cdot(\text{average }H_{\mathrm{seq}}\text{ in batch}),$$2

and, for open-ended outputs, the top-5 average of BLEU, ROUGE-1, ROUGE-L, and METEOR. Training was conducted on four NVIDIA RTX A6000 GPUs, with inference on four RTX 4090 GPUs via vLLM (Wu et al., 22 Aug 2025).

The pipeline reveals a deliberate curriculum: first align visual and textual representations, then impose explicit reasoning traces, and finally refine decision behavior through RL. A plausible implication is that the design attempts to stabilize multimodal grounding before policy optimization amplifies or suppresses particular reasoning patterns.

6. Empirical performance, qualitative behavior, and research context

On the held-out MM-Retinal-Reason test split, OphthaReason-Intern (2B) reports 83.75% accuracy on CFP in-domain basic reasoning, compared with 77.89% for FundusExpert; 73.78% on CFP out-of-domain, compared with 70.86%; 68.00% average on FFA, compared with 49.27%; and 65.06% average on OCT, compared with 44.44%. The reported gains over comparison groups are +24.92% against general MLLMs, +15.00% against medical MLLMs without RL, +21.20% against RL-based medical MLLMs, and +17.66% against ophthalmic MLLMs (Wu et al., 22 Aug 2025).

For complex reasoning, OphthaReason-Intern reports 25.35% average on the MM-Retinal-Reason complex subset, versus 21.48% for QoQ-Med, and 36.07% average on ophthalmology cases from MedXpertQA-MM, versus 27.87% for HealthGPT and 21.32% for FundusExpert. The paper also emphasizes parameter efficiency: the 2B-parameter Intern variant matches or exceeds many 7–8B models and outperforms GPT-4o on basic tasks (Wu et al., 22 Aug 2025).

The ablation study isolates the effect of staged training and UADT. Averaged across CFP, FFA, OCT, and Complex tasks, the base Qwen2.5-VL-3B scores 26.39; adding CoT SFT only yields 23.23; Stage 3 RL only yields 40.00; Stages 2 and 3 yield 46.02; Stages 1–3 yield 47.65; and Stages 1–3 with UADT yield 49.17. Heatmap analysis of advantages is reported to show that UADT amplifies positive advantages in high-entropy regions and suppresses them in low-entropy regions, consistent with adaptive allocation of learning effort (Wu et al., 22 Aug 2025).

Qualitative examples are organized to mirror clinical reasoning. In a basic CFP case, the model links microaneurysms, dot hemorrhages, and hard exudates to nonproliferative diabetic retinopathy. In a complex CFP+FFA case with gradual decrease in central vision and photopsias, it ties late-phase leakage in a subfoveal network and AMD background to neovascular wet AMD. In a history-rich case of oculocutaneous albinism, it combines congenital nystagmus, bilateral hypopigmentation, and OCT evidence of foveal hypoplasia to produce the diagnosis (Wu et al., 22 Aug 2025).

Within ophthalmic AI, MM-Retinal-Reason extends several earlier trajectories. “MM-Retinal: Knowledge-Enhanced Foundational Pretraining with Fundus Image-Text Expertise” introduced a high-quality multi-modality fundus image–text dataset and the KeepFIT foundation model for zero-shot, few-shot, and transfer learning across CFP and FFA settings (Wu et al., 2024). “M3T: Multi-Modal Medical Transformer to bridge Clinical Context with Visual Insights for Retinal Image Medical Description Generation” focused on retinal image description generation by fusing visual representations with diagnostic keywords and reported a 13.5 percentage-point absolute BLEU@4 gain over its best baseline (Shaik et al., 2024). “Multi-Modal Multi-Instance Learning for Retinal Disease Recognition” addressed selective fusion of CFP and OCT for disease recognition and emphasized lightweight interpretability-oriented multimodal aggregation (Li et al., 2021). This suggests a progression from multimodal recognition and description generation toward explicit clinical reasoning with heterogeneous textual context and supervised reasoning traces. At the same time, the absence of a reported numeric inter-annotator-agreement statistic and the fact that the model always emits full CoT traces remain important boundaries on how its reasoning claims should be interpreted (Wu et al., 22 Aug 2025).