---
title: 'FundusGen: Dataset for Ophthalmic Multimodal AI'
url: https://www.emergentmind.com/topics/fundusgen
type: topic
---

# FundusGen: Dataset for Ophthalmic Multimodal AI

FundusGen denotes, in its formal and explicit usage, the ophthalmology-specific multimodal instruction-tuning dataset constructed through the Fundus-Engine system and used to fine-tune the multimodal large language model FundusExpert. It integrates global disease classification, local object detection, fine-grained feature analysis, and clinically aligned cognitive chain reasoning on approximately 200K color fundus images, with the stated goal of enabling “positioning-diagnosis collaboration” in ophthalmic multimodal large language models [2507.17539]. In a broader interpretive sense, the name has also been used “in spirit” to describe generic or generative fundus-image systems for enhancement, synthesis, anomaly generation, modality translation, and large-scale synthetic data production [2309.00885][1706.02185][2312.16470][2312.00377].

## 1. Formal definition and system context

In the terminology of FundusExpert, FundusGen is the **dataset and instruction corpus**, not the data-construction pipeline itself. The corresponding pipeline is called **Fundus-Engine**, and the model trained on the resulting corpus is **FundusExpert**, an ophthalmology-specific MLLM based on InternVL2.5-8B [2507.17539].

| Component | Role | Main content |
|---|---|---|
| Fundus-Engine | Data construction system | Aggregation, localization, semantic expansion |
| FundusGen | Dataset and instruction corpus | ~200K images, multi-granular labels, cognitive chains |
| FundusExpert | Fine-tuned MLLM | Positioning-diagnosis collaboration |

This distinction is central. Fundus-Engine automates localization and semantic expansion; FundusGen stores the resulting multimodal supervision; FundusExpert consumes that supervision during instruction tuning. The dataset is designed to address two problems identified by the authors: fragmentation of annotation granularity and inconsistencies in clinical reasoning logic. FundusGen therefore couples region-level grounding with diagnostic reasoning sequences, rather than treating image classification, lesion localization, and report generation as unrelated tasks [2507.17539].

A plausible implication is that FundusGen should be understood less as a conventional benchmark dataset than as a structured supervision substrate for multimodal reasoning. Its unit of value is not only the image-label pair, but the joint packaging of image, region annotation, and clinically aligned textual instruction.

## 2. Data sources and annotation strata

FundusGen is built from approximately **200K color fundus images** drawn from both public and in-house sources. The public component includes **MM-Retinal**, **BRSET**, **IDRiD**, **APTOS2019**, **Messidor2**, **PAPILA**, **Retina (CataractDataset)**, and **Glaucoma_fundus**. The in-house component consists of high-quality color fundus images annotated by expert ophthalmologists with comprehensive global diagnoses and characteristic lesion labels [2507.17539].

The annotation schema is explicitly multi-granular. It combines global diseases, local structures and lesions, fine-grained visual traits, and textual descriptions intended for instruction tuning.

| Annotation layer | Content | Examples |
|---|---|---|
| Global disease labels | Multi-label diagnoses and grading | DR, DME, glaucoma, cataract, AMD |
| Local lesion/structure labels | Region-level findings | Optic disc, optic cup, hard exudates, microaneurysms, cotton-wool spots |
| Fine-grained features | Morphology and spatial distribution | Size, shape, color, clustered in macular region |
| Textual annotations | Region-grounded clinical descriptions | Objective findings and implicit diagnostic clues |

The global label space includes diabetic retinopathy with severity grading, diabetic macular edema, glaucoma, cataract, hypertensive retinopathy, age-related macular degeneration, and additional macular and rare diseases. The local annotation layer focuses on **optic disc**, **optic cup**, **hard exudates**, **microaneurysms**, and **cotton-wool spots**. Fine-grained descriptors include lesion morphology, spatial distribution, and structural characteristics such as optic disc cupping ratio and neuroretinal rim changes [2507.17539].

The textual layer is not free-form captioning in a generic sense. It is constrained to produce descriptions that are directly traceable to pixel-level or region-level evidence while also encoding clinically relevant implications. This suggests that FundusGen occupies an intermediate position between a detection dataset, a report corpus, and an instruction-tuning set.

## 3. Fundus-Engine pipeline and construction procedure

Fundus-Engine builds FundusGen through three stages: **fine-grained label collection**, **automated bounding box annotation**, and **MLLM-driven semantic expansion** [2507.17539].

The first stage aggregates approximately 200K fundus images with structured labels from open-source datasets and in-house collections. These labels include global disease information and, for selected datasets, pixel-level lesion or structure annotations. The second stage converts segmentation information into scalable localization supervision. It trains separate **nnU-Net** models for **optic cup**, **optic disc**, **hard exudates**, **microaneurysms**, and **cotton-wool spots**, using high-quality segmentation data after AutoMorph-style quality control. It then performs semi-supervised expansion by generating pseudo segmentation labels on additional images and iterating the process [2507.17539].

The paper reports cross-domain evidence for this pseudo-label strategy. For example, optic disc Dice improves from **75.3%** using true labels to **82.0%** using pseudo labels in the first round, while microaneurysm Dice improves from **21.1%** to **25.0%**. After segmentation, Fundus-Engine converts masks into boxes using **DBSCAN** with **epsilon = 160**, **min_samples = 10**, an **area > 100 pixels** filter, and retention of the **top three boxes** per lesion or structure [2507.17539].

This stage markedly expands localization density. The reported counts increase from **882** to **5,357** for microaneurysms, from **642** to **10,089** for hard exudates, from **291** to **1,876** for cotton-wool spots, from **901** to **16,551** for optic cup, and from **1,070** to **16,720** for optic disc [2507.17539]. The significance is not only scale, but the conversion of sparse segmentation resources into region-grounded supervision suitable for MLLM instruction tuning.

The third stage performs semantic expansion. Structured outputs from the first two stages are normalized, then passed through prompt templates to **GPT-4o**, which generates region-grounded text describing abnormalities, structures, and diagnostic implications. The prompting is constrained by two principles: **observational objectivity** and **clinical relevance**. Ophthalmologists then perform double-blind review, discarding or regenerating inconsistent outputs [2507.17539]. This human-in-the-loop filtering is an essential part of FundusGen’s claimed clinical alignment.

## 4. Cognitive-chain design and instruction-tuning tasks

A defining feature of FundusGen is the use of a clinically aligned **cognitive chain**, formalized as:

$$
\text{Region localization} \rightarrow \text{Feature analysis} \rightarrow \text{Diagnostic reasoning}
$$

This chain is encoded as multi-turn instruction data rather than post hoc explanation. The first class of interaction proceeds from local findings to global diagnosis: one turn asks for abnormal region analysis, and a later turn asks for a diagnostic suggestion based on those findings. The second class performs “verification and deepening of the evidence chain”: one turn gives a preliminary diagnostic analysis, and a later turn requests disease-specific evidential analysis, such as glaucoma-related optic disc and nerve fiber layer features [2507.17539].

FundusGen operationalizes this through five instruction families.

| Task type | Function | Typical output |
|---|---|---|
| General Report | Standardized ophthalmic report generation | Global diagnostic report |
| Regional QA | Localization and region labeling | Bounding boxes or grounded region text |
| Grounding Report | Report with explicit positional grounding | Region-linked narrative |
| Multi-turn Diagnostic Reasoning | Local-to-global reasoning | Findings followed by diagnosis |
| Multi-turn Confirmation Analysis | Hypothesis verification | Preliminary diagnosis followed by disease-specific evidence |

These task types jointly supervise localization, reporting, and reasoning. Regional QA directly maps coordinates to answers; Grounding Report forces text to reference image regions; the two multi-turn tasks encode temporal dependency between observations and conclusions [2507.17539].

The methodological importance of this design is that FundusGen does not treat reasoning as a purely textual add-on. It couples region grounding to dialogue structure. This suggests that, within the FundusExpert framework, interpretability is not defined only by verbal justification, but by a staged relationship between spatial evidence and diagnostic synthesis.

## 5. Empirical impact, scaling behavior, and ablation evidence

FundusGen functions as the primary supervision source for FundusExpert and also transfers effectively to other generalist MLLMs under unified instruction fine-tuning [2507.17539]. FundusExpert achieves **69.7%** accuracy on **Fundus-MMBench** and **66.7%** on the **GMAI-MMBench** fundus subset. Relative to **MedRegA (40B)**, the gains are **+29.4%** on Fundus-MMBench and **+26.6%** on GMAI-MMBench. Relative to **GPT-4o**, FundusExpert reaches **69.7 vs 41.6** on Fundus-MMBench and **66.7 vs 57.4** on GMAI-MMBench [2507.17539].

The dataset also supports strong region grounding. In zero-shot object detection evaluation, FundusExpert reaches IoU values of **0.632** for optic cup, **0.738** for optic disc, **0.194** for hard exudates, **0.141** for cotton-wool spots, and **0.116** for microaneurysms, substantially exceeding the compared MLLMs [2507.17539]. In zero-shot report generation, FundusExpert reaches **77.0%** clinical consistency versus **47.6%** for GPT-4o. The paper defines clinical consistency through semantic matching between ground-truth labels and the generated report, normalized by the union of label and semantic feature sets [2507.17539].

FundusGen also exhibits a data-quality scaling law. When FundusExpert is fine-tuned with varying fractions of FundusGen, performance on GMAI-MMBench follows:

$$
L \propto N^{0.068}
$$

with **\(R^2 = 0.972\)**, **adjusted \(R^2 = 0.930\)**, and **MSE = 0.0001** [2507.17539]. By contrast, an alternative “Classification Annotation-Guided Data” corpus, generated without localization and without explicit cognitive chains, shows no clear scaling law. The authors further report that **10% of FundusGen** yields performance comparable to **100%** of that baseline corpus, which they interpret as evidence of higher data efficiency [2507.17539].

Ablation studies indicate that FundusGen’s structure, not only its size, matters. Breaking multi-turn cognitive chains into independent single-turn tasks reduces performance from **69.7%** to **67.1%** on Fundus-MMBench and from **66.7%** to **63.2%** on GMAI-MMBench. Removing region-aware instructions lowers performance from **68.9%** to **65.3%** on Fundus-MMBench and from **64.7%** to **59.3%** on GMAI-MMBench. Removing startup General Report data lowers performance from **66.2%** to **62.9%** on Fundus-MMBench and from **60.9%** to **56.4%** on GMAI-MMBench in the reported one-epoch setting [2507.17539].

The trained FundusExpert can also replace GPT-4o inside Fundus-Engine’s semantic expansion stage. When used to generate **100K new reports** for unseen images, the resulting synthetic corpora lead to better downstream fine-tuning of **Qwen2-VL-7B** and **InternVL2.5-8B** than GPT-4o-generated data [2507.17539]. This suggests a recursive data-engineering loop in which FundusGen-trained models become domain-specific data generators.

## 6. Broader interpretive usage in fundus image generation research

Outside the strict dataset sense, “FundusGen” has also been used interpretively to denote a generic or generative fundus-image framework. This broader usage is explicit in several technical narratives, but it is not a standardized formal name across those papers.

One line of work treats FundusGen as a **generic enhancement or preprocessing block**. GFE-Net is described as pursuing “what you might call ‘FundusGen’ in spirit”: a generic, plug-and-play enhancement module trained once on clear images with synthetic degradations, without real paired low-high images, unpaired real high-quality sets, segmentations, or test-domain data during training [2309.00885]. Another line treats it as **conditional synthesis from vessel topology**. “Synthesizing Filamentary Structured Images with GANs” is described as giving “almost exactly what you would want under the name ‘FundusGen’,” with **Fila-GAN** and **Fila-sGAN** generating realistic fundus images from vessel maps and demonstrating usefulness for downstream vessel segmentation, even with as few as 10 training examples [1706.02185].

A second cluster of papers extends the idea toward **synthetic anomaly generation and trait-conditioned generation**. ReSynthDetect is presented as “exactly the kind of design relevant for a ‘FundusGen’-style framework,” combining a normal reconstruction model with a fundus-consistent anomaly generator for unsupervised anomaly detection [2312.16470]. “Futuristic Variations and Analysis in Fundus Images Corresponding to Biological Traits” is described as proposing “a prototype of what you could call ‘FundusGen’,” with **FGC-Net** generating multiple age-conditioned variants of an input fundus image [2302.03839].

A third cluster frames FundusGen as **cross-modality generation**. This includes color fundus to macular heightmap translation via conditional GANs [2009.01601], multiple families of fundus-to-fluorescein-angiography synthesis systems [2005.05267][2006.10216][2007.09191][2104.06757], and ultra-wide-angle SLO-to-FA synthesis through multi-scale GANs with attention and registration enhancement [2307.11530][2405.00542]. In these papers, the common theme is that non-invasive or lower-cost retinal modalities are used to synthesize clinically informative but invasive or expensive targets.

A fourth cluster uses the concept at dataset scale. **ReTree** employs a two-stage DDPM to generate vessel trees and then conditioned fundus images, producing **30,000** synthetic fundus-vessel pairs for vessel segmentation [2308.08339]. **SynFundus-1M** releases **over one million fundus images** with **fifteen types of annotation**, generated by a conditional diffusion model trained on more than **1.3 million** private authentic fundus images [2312.00377]. In this broader sense, FundusGen denotes a family of controllable fundus generation systems rather than a single dataset.

## 7. Limitations and future directions

FundusGen, in its formal dataset sense, has several limitations acknowledged at the system level. Disease coverage remains finite and is restricted to **fundus photography**; other ophthalmic modalities such as OCT and fundus autofluorescence are not included. The data sources span different countries and devices, but this also introduces potential **demographic and device bias**. Region annotations depend partly on pseudo-labels produced by **nnU-Net**, and the reported segmentation quality is imperfect for small lesions such as microaneurysms and cotton-wool spots. The textual layer initially depends on **GPT-4o**, which can introduce subtle semantic bias or hallucination despite expert review. Finally, the clinical cognitive chain itself is hand-crafted and may reflect a particular reasoning style rather than the full diversity of ophthalmic diagnostic practice [2507.17539].

The future directions proposed around FundusGen are correspondingly systemic. The authors point to **reinforcement learning on FundusGen’s semantic hierarchy**, **dynamic reasoning and test-time scaling**, extension of Fundus-Engine to **other ophthalmic modalities and clinical scenarios**, and expansion of pseudo-labeling to more lesions and structures [2507.17539]. A plausible implication is that FundusGen may evolve from a fundus-only instruction corpus into a multimodal ophthalmic data engine with region grounding, structured reasoning, and domain-specific synthetic annotation loops.

In the broader interpretive literature, related future directions include stronger structure-preserving enhancement backbones, more advanced frequency modeling, learned lesion generators, diffusion-based refinement, style disentanglement, and broader cross-modality generation [2309.00885][2312.16470]. Taken together, these trajectories suggest that “FundusGen” now names both a specific dataset in ophthalmic MLLM research and a wider design ambition: a clinically aligned, controllable, and scalable generative infrastructure for fundus-centered ophthalmic AI.

Source: https://www.emergentmind.com/topics/fundusgen