FundusExpert: Ophthalmic Cognitive LLM

Updated 4 July 2026

FundusExpert is an ophthalmology-specific multimodal LLM that integrates local object localization, fine-grained feature analysis, and global diagnostic reasoning into a unified clinical cognitive chain.
It leverages FundusGen—a dataset with over 200K fundus images and multi-granularity annotations—to resolve fragmented labeling and reasoning inconsistencies.
The model demonstrates enhanced clinical performance with robust scaling, precise region grounding, and traceable evidence-based outputs.

FundusExpert is an ophthalmology-specific multimodal LLM with integrated positioning-diagnosis reasoning capabilities, introduced together with FundusGen and the Fundus-Engine data system to address two recurrent problems in fundus AI: fragmented annotation granularity and inconsistencies in clinical reasoning logic. Its central design principle is to connect local object localization, fine-grained feature analysis, and global disease classification within a clinically aligned cognitive chain, so that fundus interpretation proceeds from region-level evidence to diagnostically grounded conclusions rather than from isolated labels alone (Liu et al., 23 Jul 2025).

1. Conceptual basis and diagnostic scope

FundusExpert was proposed against a backdrop in which general and medical multimodal LLMs remained weak on core fundus-reading skills. The motivating diagnosis is not merely that performance was suboptimal, but that supervision itself was fragmented: many datasets offered only global disease labels, while others isolated local annotations such as optic disc or lesion masks, leaving the clinically important transition from localized evidence to global diagnosis under-specified. FundusExpert responds by making “positioning-diagnosis collaboration” a first-class modeling target, so that localization is not an auxiliary visualization step but part of the diagnostic computation (Liu et al., 23 Jul 2025).

This design choice is consistent with benchmark evidence showing that existing MLLMs often fail at even basic ophthalmic perception. FunBench organizes 91,810 visual questions across four levels—modality perception, anatomy perception, lesion analysis, and disease diagnosis—and reports “significant deficiencies in fundus reading skills, particularly in basic tasks such as laterality recognition.” Its rank-correlation analysis further shows that current MLLM ranks track their LLM-only ranks far more than their VE-only ranks, with Spearman $0.917$ versus $0.055$, indicating that many systems remain dominated by language-side priors rather than robust fundus-specific visual grounding (Wei et al., 2 Mar 2025).

2. FundusGen and the Fundus-Engine annotation pipeline

FundusExpert is trained on FundusGen, a corpus built by Fundus-Engine to fuse global disease labels, local object localization, and fine-grained feature descriptions on the same fundus image. FundusGen aggregates approximately 200K fundus images from open-source and in-house sources, while the instruction-tuning stage uses roughly 300,000 instruction samples and is kept strictly isolated from Fundus-MMBench during evaluation. The data include global labels such as diabetic retinopathy and diabetic macular edema grading, region annotations for optic disc, optic cup, hard exudates, microaneurysms, and cotton-wool spots, and semantically expanded feature descriptions validated by clinicians (Liu et al., 23 Jul 2025).

Fundus-Engine combines two automated stages. First, a category-specific nnU-Net is trained for five representative targets—optic disc, optic cup, hard exudates, microaneurysms, and cotton-wool spots—and expanded through semi-supervised self-training with pseudo-labels. Second, those segmentations are converted to bounding boxes using DBSCAN with $\text{epsilon}=160$ and $\text{min\_samples}=10$ ; candidate boxes are filtered by area $>100$ pixels and the top-3 by area are retained. In parallel, an MLLM-based semantic expansion stage generates clinically standardized descriptions that bind coordinates, lesion semantics, and diagnostic implications, after which double-blind expert review filters or regenerates substandard outputs (Liu et al., 23 Jul 2025).

Component	Function	Reported details
Fundus-Engine	Localization and semantic expansion	nnU-Net for OD, OC, EX, MA, CWS; DBSCAN $\text{epsilon}=160$ , $\text{min\_samples}=10$
FundusGen	Instruction corpus	Approximately 200K images; roughly 300,000 instruction samples
Instruction types	Multi-granularity supervision	General Reports, Regional QA, Grounding Reports, Multi-turn Diagnostic Reasoning, Multi-turn Confirmation Analysis

The scale expansion achieved by pseudo-labeling is substantial. Reported instance counts increase from 1,070 true OD labels to 16,720 pseudo-labels, from 901 true OC labels to 16,551 pseudo-labels, from 642 true EX labels to 10,089 pseudo-labels, from 882 true MA labels to 5,357 pseudo-labels, and from 291 true CWS labels to 1,876 pseudo-labels. This suggests that FundusExpert’s localization capability is built not only from manually curated annotations but from an explicitly engineered pseudo-labeling regime designed to make region-grounded supervision scalable (Liu et al., 23 Jul 2025).

3. Architecture and clinical cognitive chain reasoning

FundusExpert is implemented by fully fine-tuning an InternVL2.5-8B base model. Its backbone comprises an InternViT vision encoder with approximately 300M parameters, an MLP-based multimodal connector, and an InternLM language backbone with approximately 7B parameters, for a total parameter scale of approximately 8B. A 1B deployment-oriented variant, FundusExpert-mini, is also provided for efficiency-oriented use cases (Liu et al., 23 Jul 2025).

A notable aspect of the architecture is that it does not introduce a separate detection head. Instead, localization is textualized inside the instruction format. Region-aware capabilities are learned through tasks such as “Regional QA” and “Grounding Report,” where bounding boxes are encoded as coordinates in text and tied to lesion descriptions and final diagnoses in multi-turn dialogues. This means that localization, feature description, and diagnostic reasoning are optimized under a unified autoregressive objective rather than separated into detection and classification subnets. The paper presents a conceptual decomposition,

$L_{\text{total}} = L_{\text{cls}} + L_{\text{loc}} + L_{\text{reason}} + \lambda L_{\text{align}},$

but states that in practice all supervision is realized as next-token negative log-likelihood on multimodal instructions and responses (Liu et al., 23 Jul 2025).

The model’s cognitive-chain supervision mirrors ophthalmic workflow. One pattern moves from local to global: a first round asks the model to analyze abnormal regions, and a second round asks for a diagnostic suggestion conditioned on those findings. Another pattern begins with a preliminary diagnosis and then requests targeted verification, such as analysis of features suggesting glaucoma. These multi-turn sequences are intended to encode “clinical cognitive chain reasoning,” so that diagnostic outputs remain traceable to localized evidence instead of being generated as unsupported labels (Liu et al., 23 Jul 2025).

4. Evaluation, benchmark behavior, and scaling

FundusExpert is evaluated on Fundus-MMBench and the fundus subset of GMAI-MMBench for clinical question answering, on zero-shot localization using IoU, and on zero-shot report generation using a clinical consistency score defined by multi-granularity semantic matching. On Fundus-MMBench, FundusExpert achieves $69.7\%$ ; on GMAI-MMBench, it achieves $66.7\%$ , surpassing MedRegA-40B by $0.055$0 on GMAI-MMBench. In zero-shot report generation, its clinical consistency reaches $0.055$1, compared with GPT-4o’s $0.055$2 (Liu et al., 23 Jul 2025).

Evaluation aspect	Reported result	Significance
Fundus-MMBench	69.7%	In-distribution clinical QA
GMAI-MMBench	66.7%	Out-of-distribution clinical QA
Against MedRegA-40B	+26.6% on GMAI-MMBench	Stronger than a larger medical baseline
Zero-shot report generation	77.0% clinical consistency	GPT-4o baseline: 47.6%
Zero-shot localization	OD 0.738, OC 0.632, EX 0.194, CWS 0.141, MA 0.116	Region grounding quality
Scaling law	$0.055$3	$0.055$4, adjusted $0.055$5, MSE $0.055$6

The localization results indicate markedly different levels of difficulty across structures and lesions. Optic disc and optic cup localization achieve IoU values of $0.055$7 and $0.055$8, while hard exudates, cotton-wool spots, and microaneurysms achieve $0.055$9, $\text{epsilon}=160$ 0, and $\text{epsilon}=160$ 1, respectively. This disparity is consistent with the clinical fact that disc- and cup-level structures are larger and more spatially coherent than sparse, small, or morphologically variable lesion targets. The report-generation result is also important because the evaluation is not phrased as raw text overlap; it is phrased as “clinical consistency,” which better matches the intended use of diagnostic reporting (Liu et al., 23 Jul 2025).

The scaling-law analysis is central to the paper’s argument about annotation quality. On GMAI-MMBench, subset sampling reveals

$\text{epsilon}=160$ 2

with $\text{epsilon}=160$ 3, adjusted $\text{epsilon}=160$ 4, and MSE $\text{epsilon}=160$ 5. The reported interpretation is that cognitively aligned, region-grounded supervision is more data-efficient than classification-only supervision. A related operational point is that FundusExpert-mini reaches $\text{epsilon}=160$ 6 img/s at $\text{epsilon}=160$ 7 GB VRAM with batch size 1 on an RTX 4090, scaling to $\text{epsilon}=160$ 8 img/s at maximum batch size 128, while achieving $\text{epsilon}=160$ 9 on Fundus-MMBench and $\text{min\_samples}=10$ 0 on GMAI-MMBench (Liu et al., 23 Jul 2025).

5. Position within fundus foundation-model and MLLM research

FundusExpert belongs to a broader progression in fundus AI from image-only pretraining to knowledge-rich vision-language learning and finally to reasoning-centric MLLMs. Earlier retina foundation models emphasized explicit expert knowledge at the prompt level. FLAIR, for example, compiled 38 open-access fundus datasets with up to 101 target conditions and 288,307 images, and encoded expert knowledge through descriptive textual prompts during both pretraining and zero-shot inference, showing strong domain-shift and unseen-category behavior relative to generalist VLMs (Silva-Rodríguez et al., 2023).

A second line of work enriched the textual side itself. MM-Retinal assembled 4,349+ high-quality image-text pairs from professional fundus diagram books and introduced KeepFIT, which uses image similarity-guided text revision and mixed training to transfer expert fundus descriptions into foundational pretraining. KeepFIT trained on flair + MM-Retinal reached an average of $\text{min\_samples}=10$ 1 across six unseen tasks, outperforming KeepFIT trained on flair only and FLAIR-based baselines in the reported transfer setting (Wu et al., 2024).

A third line expanded disease coverage and zero-shot clinical breadth. RetiZero pretrained on 341,896 fundus image-text pairs covering more than 400 fundus diseases and reported zero-shot Top-5 accuracies of $\text{min\_samples}=10$ 2 on EYE-15 and $\text{min\_samples}=10$ 3 on EYE-52, while its image-to-image retrieval Top-5 scores reached $\text{min\_samples}=10$ 4 and $\text{min\_samples}=10$ 5 on the same sets. In the same general direction, Fundus-R1 demonstrated that a reasoning-enhanced fundus-reading MLLM could be trained exclusively from public data: it used 168,938 images, of which over 94% had only image-level labels, and coupled RAG-based reasoning traces with RLVR and a process reward that enforced self-consistency in the generated reasoning trace (Wang et al., 2024, Deng et al., 9 Apr 2026).

This progression suggests a structural shift in the field. Earlier systems established expert prompt engineering and retinal image-text alignment; later systems expanded disease breadth and retrieval; FundusExpert moves further by treating localization, evidence synthesis, and diagnosis as a single instruction-tuned reasoning problem rather than as adjacent modules. That distinction is not merely architectural. It changes what counts as supervision: not only labels or prompts, but clinically ordered reasoning paths (Liu et al., 23 Jul 2025).

6. Preconditions, limitations, and system-level implications

Work framed around FundusExpert repeatedly treats image quality and enhancement as prerequisites for reliable reasoning. GFE-Net, an annotation-free fundus enhancement network using frequency self-supervised representation learning, is trained from a small clear-image set and generalizes from 40 clear DRIVE images to FIQ, EyeQ, RCF, Kaggle, and Fundus-iSee without test-time adaptation. It reports FIQ SSIM $\text{min\_samples}=10$ 6 and PSNR $\text{min\_samples}=10$ 7, and on Fundus-iSee cataract images its enhancement raises disease-diagnosis performance from F1 $\text{min\_samples}=10$ 8/Ckappa $\text{min\_samples}=10$ 9 to F1 $>100$ 0/Ckappa $>100$ 1 (Li et al., 2023).

Quality assessment has likewise been formalized as a dedicated layer. FTHNet, trained on the FQS dataset of 2,246 clinically acquired fundus images, predicts a continuous Mean Opinion Score and reports PLCC $>100$ 2 and SRCC $>100$ 3 in the abstract, while FundaQ-8 proposes an eight-parameter clinically inspired scoring framework with

$>100$ 4

where each $>100$ 5 and thresholds are set as Bad: score $>100$ 6, Medium: $>100$ 7, and Good: score $>100$ 8 (Gong et al., 2024, Zun et al., 25 Jun 2025).

More recent work pushes this further toward explainability. FunPiQ introduces pixel-level quality annotations and argues that pixel-level FIQA based on anatomical visibility is a more task-agnostic and explainable formulation than image-level labels; its EFIQA-CP method achieves a merged-set QWK of $>100$ 9, mDice of $\text{epsilon}=160$ 0, Reject Dice of $\text{epsilon}=160$ 1, AUROC of $\text{epsilon}=160$ 2, AUPRC of $\text{epsilon}=160$ 3, and S95 of $\text{epsilon}=160$ 4 (Wang et al., 24 Jun 2026).

The limitations of FundusExpert itself remain substantial. The current automated localization pipeline focuses on only five representative targets—OD, OC, EX, MA, and CWS—so other clinically important findings such as hemorrhage subtypes, neovascularization, or drusen are less explicitly supervised. The system also relies on pseudo-labels and MLLM-based semantic expansion, which introduces residual noise even after filtering. Domain shift across devices, acquisition protocols, and populations remains a deployment concern, and the paper explicitly positions FundusExpert as a decision-support tool rather than a replacement for clinical judgment (Liu et al., 23 Jul 2025).

A common misconception is that increasingly large generalist MLLMs can simply be adapted to fundus imaging with enough generic vision-language data. The accumulated evidence does not support that view. FunBench shows persistent weaknesses in laterality and other elementary anatomical tasks, while FundusExpert’s own results indicate that gains come from clinically aligned cognitive chains and region-grounded supervision rather than parameter count alone. A plausible implication is that future progress in fundus AI will depend less on generic scaling than on the quality of ophthalmic alignment between localized evidence, expert terminology, and diagnostic reasoning (Wei et al., 2 Mar 2025).