EyeMoSt t+: Evidential Multimodal Eye Screening

• EyeMoSt t+ is an evidential multimodal deep learning pipeline that fuses 2D fundus images and 3D OCT volumes to provide robust, calibrated predictions for eye disease screening.
• The pipeline employs a principled mixture of Student’s t-distributions to adaptively fuse modality-specific predictions and quantify both local and global uncertainties.
• Its adaptive design enhances screening accuracy under noisy conditions while acting as a data quality discriminator in clinical decision support.

EyeMoSt is an evidential multimodal deep learning pipeline designed for reliable eye disease screening by integrating 2D fundus photographs and 3D optical coherence tomography (OCT) volumes. It addresses the challenge of unreliable or noisy data in one or more modalities by adaptively fusing modality-specific predictions using a principled mixture of Student's t-distributions (MoSt). EyeMoSt provides calibrated classification predictions along with both local (per modality) and global (fused) uncertainty quantification, and serves as a data quality discriminator in multimodal ophthalmic decision support (Zou et al., 2023).

1. Motivation and Background

Multimodality in ophthalmologic screening leverages complementary data—commonly pairing 2D fundus images and 3D OCT scans—to improve disease detection such as glaucoma grading and AMD vs. PCV discrimination. Classical fusion methods, typically implemented with early or intermediate-fusion convolutional neural networks, treat each input modality equivalently. This neglects potential modality unreliability due to noise, incompleteness, or distributional shift, risking degraded screening performance. Reliable estimation of both unimodal and fusion-level prediction uncertainty is critical for safety and clinical robustness. EyeMoSt is designed to provide principled confidence measures for each modality and for the joint outcome (Zou et al., 2023).

2. Model Architecture and Workflow

The EyeMoSt pipeline consists of sequential stages for multimodal encoding, evidential uncertainty estimation, distributional fusion, and decision-making:

• Feature Extraction:
  • Fundus images ($x_1$) are encoded via a 2D Res2Net CNN.
  • OCT volumes ($x_2$) are processed with a 3D MedicalNet CNN.
  • Both produce learned feature vectors.
• Evidential Heads:
  • For each modality $m \in \{1, 2\}$, a modality-specific MLP ("evidential head") outputs the Normal-Inverse-Gamma (NIG) parameters $$\mathbf{p}_m = (\gamma_m,\delta_m,\alpha_m,\beta_m)$$ representing the evidential distribution over the regression head.
• Predictive Student’s t Distributions:
  • Marginalization of the Gaussian likelihood (parameterized by mean $\mu$ and variance $\sigma^2$) under the NIG prior yields a predictive Student’s t-distribution per modality: $\mathrm{St}(y; u_m, \Sigma_m, \nu_m)$.
• Adaptive Fusion via Mixture-of-Student’s t:
  • The two unimodal Student's t-distributions are fused into a single distribution, selecting the smaller degrees of freedom (heavier-tailed for robustness), and computing a weighted covariance structure.
  • The fused prediction yields both a point estimate and a global uncertainty score.

The schematic workflow is encapsulated in the following high-level pseudocode:

$$\mathbf{p}_m = (\gamma_m,\delta_m,\alpha_m,\beta_m)$$1

3. Evidential Uncertainty Quantification

Each modality's prediction $y$ is modeled as drawn from a Gaussian $y \sim \mathcal{N}(\mu, \sigma^2)$, where the parameters $\mu$ and $x_2$0 possess an evidential NIG prior:

$x_2$1

The evidential head predicts the tuple $x_2$2, from which uncertainty is decomposed as:

• Aleatoric uncertainty (data noise):

$x_2$3

• Epistemic uncertainty (model confidence):

$x_2$4

Both are estimated per modality in a single forward pass, informing the fusion process.

4. Fusion via Mixture of Student's t Distributions

The fusion mechanism constructs a robust global prediction by combining unimodal Student’s t distributions:

• Unimodal Kernel:

Each predictive density is

$x_2$5

where the closed-form PDF in $x_2$6 dimensions is

$x_2$7

• Mixture Construction:

The fused density for $x_2$8 is

$x_2$9

where mixture weights $m \in \{1, 2\}$0 (non-negative, sum to 1) correspond to evidential mass and are implicitly derived from NIG parameters.

• Adaptive Degrees of Freedom:

Heavier tails (i.e., smaller $m \in \{1, 2\}$1) reflect increased robustness against modality noise or outliers. EyeMoSt sets

$m \in \{1, 2\}$2

ensuring fusion is dominated by the most robust (heaviest-tailed) individual view.

5. Training and Loss Formulation

Training is performed by minimizing the sum of loss terms for modality-specific and fused predictions:

$m \in \{1, 2\}$3

where, for each $m \in \{1, 2\}$4,

$m \in \{1, 2\}$5

with $m \in \{1, 2\}$6 and CE the cross-entropy loss on the softmax of $m \in \{1, 2\}$7. The fused loss uses the Student's t density for the combined output. Optimization uses Adam ($m \in \{1, 2\}$8, no weight decay), batch size 16, for up to 100 epochs with early stopping.

6. Empirical Performance and Robustness

EyeMoSt was evaluated on two datasets:

• GAMMA: 100 Fundus–OCT pairs (three-level glaucoma grading, five-fold cross-validation)
• In-house: 604 Fundus–OCT pairs (AMD vs. PCV)

Metrics included Accuracy (ACC), Cohen’s Kappa, AUC for classification, Expected Calibration Error (ECE) for calibration, and robustness to additive Gaussian noise ($m \in \{1, 2\}$9) in a single modality.

Key findings:

Scenario	EyeMoSt ACC/Kappa	Baseline ACC (noisy OCT)	EyeMoSt ACC (noisy OCT)	EyeMoSt ECE
Clean (GAMMA)	0.85 / 0.75	—	—	0.17
Noise, $$\mathbf{p}_m = (\gamma_m,\delta_m,\alpha_m,\beta_m)$$0 (OCT)	—	~0.53	~0.83	0.17–0.20 (vs. 0.20–0.45 baseline)

On clean data, EyeMoSt achieved top-tier accuracy and Kappa, with significantly better robustness under noise: baseline methods showed a marked drop in ACC (to ~0.53), while EyeMoSt retained ~0.83. EyeMoSt's calibration error was consistently lower than alternatives. Heavy-tailed fusion directly contributed to this noise resilience.

7. Data Quality Discrimination and Outlier Detection

EyeMoSt detects and down-weights unreliable modalities using its evidential uncertainty signals:

• Noising one modality raises its local uncertainty (epistemic + aleatoric), as measured by its NIG head.
• The fusion rule shifts to favor the cleaner modality’s Student’s t (with heavier tails, lower uncertainty).
• Empirically, the fused uncertainty distribution under noise conditions remains closer to the clean modality’s distribution, demonstrating adaptive trust in data quality.
• EyeMoSt thus functions as a data-quality discriminator or out-of-distribution detector, automatically mitigating the influence of untrustworthy inputs, which supports reliable and safe multimodal clinical decision-making.

8. Conclusion and Significance

EyeMoSt introduces a principled evidential fusion mechanism for multimodal eye disease screening, combining calibrated prediction, robust uncertainty quantification, and data quality discrimination, delivered through theoretically grounded mixture-of-Student’s t distributions. This approach yields superior accuracy, calibration, and robustness to input corruption relative to conventional multimodal fusion pipelines, as established through extensive evaluation (Zou et al., 2023).