Meta-EyeFM: AI for Ocular Diagnostics

• Meta-EyeFM is a multi-modal foundation model that integrates a large language model with vision models to analyze fundus images and deliver conversational diagnostic support.
• It employs a modular architecture with LoRA-based fine-tuning and an advanced routing mechanism to achieve high accuracy in disease detection, severity grading, sign identification, and systemic prediction.
• The system outperforms top multimodal models and rivals expert ophthalmologists, offering scalable, data-efficient, and clinically relevant solutions for ocular diagnostics.

Meta-EyeFM is a multi-function foundation model integrating a LLM with a suite of vision foundation models (VFMs) for advanced ocular disease assessment from fundus images. This system is designed to deliver user-friendly, conversational diagnostic support to clinicians through sophisticated query routing and expert-level vision analysis. By employing modular architecture, LoRA-based fine-tuning, and rigorous task-specific heads, Meta-EyeFM achieves high diagnostic accuracy across disease detection, severity grading, sign identification, and systemic disease prediction tasks, with demonstrated performance surpassing leading large multimodal models and equaling senior ophthalmologists in clinical benchmarking (Soh et al., 13 May 2025).

1. Model Structure and System Components

Meta-EyeFM comprises three primary modules:

• A multi-modal LLM “router” based on LLaVA with LoRA.
• The Meta-Eye Router, a gating MLP responsible for expert selection.
• A pool of eight task-specific VFMs, each specializing in one of: ocular disease detection (diabetic retinopathy [DR], age-related macular degeneration [AMD], myopic macular degeneration [MMD], cataract, glaucoma); disease severity grading; ocular sign identification; systemic disease prediction (diabetes, hypertension, chronic kidney disease).

The system processes a natural-language query $X_t$ and a fundus image $X_{img}$, which are passed to the LLM. The LLM generates an output sequence $y$ containing a special <Router> token. The embedding $h_{router}$ (derived from this token) is extracted from the LLM’s final layer. This embedding feeds into the Meta-Eye Router (a two-layer MLP), which outputs routing probabilities $y_{router} \in \mathbb{R}^D$ ($D$ being the number of VFMs). The expert VFM is selected as $VFM_i = \arg\max_i\; y_{router,i}$, to which the raw image and query context are forwarded. Each VFM consists of a ViT-MAE encoder and a task-specific head, returning the appropriate diagnostic output.

2. Routing Methodology and Gating Mechanism

The routing strategy follows an “embedding-as-router” paradigm. The LLM encodes both textual and visual input, producing a token sequence that includes <Router>. The last-layer embedding $h_{router}$ of this token is projected through a two-layer MLP, with the mapping:

$$\ell = W_2(\mathrm{GELU}(W_1h_{router} + b_1)) + b_2$$

Routing probabilities are obtained using either sigmoidal (for multi-label) or softmax (for multi-class) activations:

$$y_{router,i} = \sigma(\ell_i), \quad \mathrm{or} \quad \mathrm{softmax}(\ell)_i = \frac{\exp(\ell_i)}{\sum_{j=1}^D \exp(\ell_j)}$$

The model applies hard routing with:

$X_{img}$0

During LLM fine-tuning, expert selection accuracy is enforced using a binary cross-entropy loss:

$X_{img}$1

where $X_{img}$2 is the one-hot label for the correct VFM. This yields 100% accuracy in expert selection on internal validation.

3. LoRA-Based Fine-Tuning and Data Efficiency

LoRA (Low-Rank Adaptation) adapters are inserted into both the LLM and each VFM, freezing most pre-trained parameters and adapting only a small number of rank-constrained layers. Parameter updates for a pre-trained weight $X_{img}$3 are modeled as:

$X_{img}$4

with $X_{img}$5, $X_{img}$6, $X_{img}$7. Only $X_{img}$8 and $X_{img}$9 are updated during fine-tuning.

The overall loss combines text modeling and routing objectives:

$y$0

LoRA-based adapters, by enabling data-efficient adaptation, allow extension to new diseases or camera types using limited additional annotations.

4. Performance Evaluation and Metrics

Meta-EyeFM demonstrates the following performance on internal test sets:

Task	Min Accuracy	Mean/Overall Accuracy	Max Accuracy	Min AUC	Mean/Overall AUC	Max AUC
Disease detection	82.2%	88.1% (cataract)	94.0%	91.2%	95.7%	98.8%
Severity grading	82.5%	90.8%	97.5%	82.0%	94.9%	99.2%
Sign identification	76.3%	88.9%	96.4%	77.9%	91.7%	99.0%
Systemic prediction	73.8%	74.4%	78.5%	79.8%	81.8%	84.8%

Key results:

• Routing accuracy is 100%.
• Disease detection achieves ≥82.2% accuracy and AUC≥91.2%.
• Severity grading achieves ≥82.5% accuracy and AUC≥82.0%.
• Sign identification achieves ≥76.3% accuracy and AUC≥77.9%.
• Systemic prediction achieves ≥73.8% accuracy and AUC≥79.8%.

External test sets display comparable generalizability (AUC 81–100% for ocular tasks and 65–94% accuracy across domains).

5. Comparative Analysis with State-of-the-Art Models

Meta-EyeFM’s external performance was benchmarked against Gemini-1.5-flash and ChatGPT-4o across four ocular diseases, with the prompt: “What disease is in this image?” (100 cases, 100 controls per disease):

Disease	Meta-EyeFM	Gemini	GPT-4o
DR (IDRID)	86.5%	65.5%	73.0%
AMD (PIONEER)	76.0%	49.5%	51.0%
MMD (CGMH)	91.5%	50.5%	50.0%
Glaucoma (PIONEER)	80.5%	48.5%	50.5%

Meta-EyeFM outperforms Gemini by 21–41% and GPT-4o by 13.5–41.5%, with all performance improvements reaching $y$1 (McNemar’s test). In clinical benchmarking on SEED (n=60 images, 5-class), Meta-EyeFM achieves $y$2, on par with the senior ophthalmologist ($y$3) and above other ophthalmologists ($y$4) and optometrists ($y$5). The hardest class is glaucoma (Meta-EyeFM $y$6 vs. expert $y$7).

6. Clinical Usage Patterns and Deployment

Meta-EyeFM functions as a conversational decision-support and triaging interface, tailored for primary care and tele-ophthalmology. Users upload a fundus image and a free-text query (e.g., “Is there any DR?”). The system parses intent, routes to the correct VFM, and returns results such as binary disease detection, confidence scores, severity grades, key sign identification, and recommended next steps (e.g., “Refer to ophthalmology for OCT and fluorescein angiography”).

Follow-up capabilities include:

• Routing to a severity-grading VFM on severity queries (e.g., “How severe is the AMD?”).
• Routing to a sign-identification VFM for questions about features (e.g., “What signs do you see?”).
• Routing to a systemic-prediction VFM for systemic disease queries (“Any sign of diabetes?”).

Key deployment options encompass both local servers/cloud APIs in secure hospital environments and web or smartphone-based portals for remote screening. Notable features include the text-based UI, granular outputs, modular extensibility, and deterministic routing correctness.

7. Significance and Future Implications

Meta-EyeFM inaugurates an integrated LLM-VFM framework for foundation model-driven, fundus-based diagnostics. Its architecture modularizes expert knowledge across disease categories and tasks, providing high accuracy, data-efficient adaptation (via LoRA), and expert-level equivalence in clinical practice. The modular-gated design offers extensibility toward additional diseases and supports scalable deployment models. A plausible implication is broader application in conversational diagnostic systems and the expansion of multi-expert routing paradigms in medical AI (Soh et al., 13 May 2025).