MultiRetNet: DR Staging & Deferral System

• The paper introduces a multimodal diabetic retinopathy staging system combining a fine-tuned ResNet18 with selected patient metadata to enhance diagnostic accuracy.
• It employs a fully connected fusion strategy to create joint embeddings and implements a deferral mechanism that routes low-quality or uncertain cases to clinicians.
• Empirical results demonstrate improved accuracy and reduced dangerous misclassifications, highlighting the system’s potential for safe, real-world screening.

Searching arXiv for MultiRetNet and related papers to ground the article. {"query":"MultiRetNet diabetic retinopathy arXiv (She et al., 19 Jul 2025)", "max_results": 5} MultiRetNet is a multimodal diabetic retinopathy (DR) staging system that combines retinal fundus images with non-image patient metadata and augments the predictor with a deferral mechanism so that uncertain or out-of-distribution cases can be routed to a clinician. In the cited 2025 work, the term denotes a two-part pipeline: a five-class DR staging model built from a fine-tuned ResNet18 image encoder plus selected tabular variables, and a human-in-the-loop quality-control subsystem called ContrastiveNet for clinician deferral on poor-quality inputs (She et al., 19 Jul 2025). The name also requires disambiguation: in the 2020 semantic-segmentation paper “Multi Receptive Field Network for Semantic Segmentation,” “MultiRetNet” is only an informal or alternate shorthand for the proposed Multi Receptive Field Network and is not the official name used by the authors (Yuan et al., 2020).

1. Nomenclature and scope

The principal meaning of MultiRetNet in the current arXiv record is the multimodal DR staging and deferral system introduced in 2025 (She et al., 19 Jul 2025). Its stated objective is to improve staging accuracy by integrating retinal imaging with socioeconomic, demographic, and diabetes-history variables, while adding a clinical deferral mechanism for cases that should not be trusted to the automated model alone. The motivating deployment setting is explicitly realistic rather than idealized: handheld retinal cameras, limited screening access, and variable image quality.

A recurrent source of confusion is terminological rather than architectural. The 2020 paper “Multi Receptive Field Network for Semantic Segmentation” proposes a semantic-segmentation architecture centered on a Multi-Receptive Field Module (MRFM) and an edge-aware loss, but the paper itself does not use the name “MultiRetNet” (Yuan et al., 2020). Accordingly, the DR system and the semantic-segmentation network are distinct works with different tasks, backbones, losses, and evaluation protocols. This suggests that “MultiRetNet” should be treated as a task-specific proper name in the 2025 DR context, not as a generic label for multi-scale or retentive vision models.

2. Clinical motivation, data, and preprocessing

The DR formulation is a five-stage classification problem over retinal fundus images enriched with tabular patient context. The cited work argues that retinal appearance alone does not fully capture disease risk or delayed presentation, and therefore treats selected metadata as clinically meaningful context that can complement image evidence, particularly when staging boundaries are visually subtle or images are suboptimal (She et al., 19 Jul 2025).

The experiments use the mBRSET dataset, described as a public retinal dataset collected using portable handheld cameras in real-world settings. After removing samples with unknown DR stage labels, the retained corpus contains 4,884 images from 1,287 patients. Missing values in dm_time and educational_level affecting about 1% of patients are imputed using the mode. Tabular variables are standardized to zero mean and unit variance with StandardScaler. Images are cropped to $224 \times 224$ and normalized using ImageNet channel statistics,

$$\mu = [0.485, 0.456, 0.406], \qquad \sigma = [0.229, 0.224, 0.225].$$

Two data-partitioning regimes are used. For unimodal baselines, samples are split randomly into 80% train and 20% test. For multimodal fusion experiments, the study additionally downsamples to a class-balanced dataset with 82 images per class and uses 5-fold cross-validation together with a held-out balanced test set. The paper also notes that the original dataset is highly imbalanced, with 76.7% of examples in Class 0. That imbalance shapes the training design, especially for the image-only baseline.

3. Architecture and multimodal fusion

The image branch is built from a pretrained ResNet18. It is first fine-tuned as an image-only DR classifier by replacing the original final layer with a 5-way classifier corresponding to the five DR stages. Because of class imbalance, training uses weighted cross-entropy with class weights

$[0.2605,\; 3.5912,\; 1.7197,\; 11.9122,\; 4.6075].$

This model is trained for 20 epochs with Adam at learning rate $10^{-3}$. For multimodal training, the fine-tuned ResNet18 is reused with its final two layers removed, yielding 512-dimensional image embeddings, and the ResNet-derived weights are then frozen so that only the fusion and classification layers are updated (She et al., 19 Jul 2025).

The tabular branch begins from 17 tabular variables described as including socioeconomic factors, demographics, and clinical information. A simple tabular neural network is trained with a 17-dimensional input, one hidden layer of 32 units with ReLU, and a 5-logit output layer, also for 20 epochs with Adam at learning rate $10^{-3}$ and class-weighted cross-entropy. The authors then compute SHAP values and select four variables for multimodal modeling: educational level, sex, time since diabetes diagnosis (dm_time), and age. Although the paper motivates multimodal modeling partly with socioeconomic variables and comorbidity profiles, the actual fusion experiments use these four selected tabular features.

Each scalar tabular feature is encoded with a shared projection layer consisting of a linear transformation followed by layer normalization, producing 512-dimensional tabular embeddings in the same latent dimensionality as the image embeddings. The paper evaluates three fusion strategies:

• Concatenation: mean-pool image and tabular embeddings separately, then concatenate them.
• Fully connected layer fusion: apply a learned linear layer to the pooled-and-concatenated representation.
• Cross-attention: use bidirectional cross-attention so that image embeddings attend to tabular embeddings and tabular embeddings attend to image embeddings.

For multimodal training, the downsampled balanced dataset is used, runs last 10 epochs, optimization again uses Adam at $10^{-3}$, and only the fusion and classifier weights are updated. The study ultimately describes fusion through a fully connected layer as the “most versatile” methodology, not because it dominates the staging metrics, but because it produces the most useful joint representation for the downstream deferral system (She et al., 19 Jul 2025).

4. Deferral system and human-in-the-loop operation

A defining component of MultiRetNet is its explicit deferral mechanism, which is intended to determine when a sample should be accepted by the automated pipeline and when it should be routed to clinician review. The paper motivates this by the prevalence of poor-quality fundus images in low-resource or handheld-camera settings and frames the mechanism as a safety-oriented response to out-of-distribution or visually degraded inputs rather than as a second disease-stage predictor (She et al., 19 Jul 2025).

To create training data for deferral, the authors synthesize low-quality “adversarial” images. They randomly select 20 images from each DR stage, yielding 100 total source images, and apply three transformations intended to mimic realistic acquisition problems: random rotation in the range $-30^\circ$ to $30^\circ$, Gaussian blur with kernel size 5, and random color jitter with maximum brightness, contrast, and saturation adjustments of 0.2. This produces 100 adversarial images, balanced across the five DR classes. In the paper’s framing, out-of-distribution cases are therefore primarily degraded versions of in-distribution retinal images rather than external-domain pathology or device-shift samples.

The deferral model, ContrastiveNet, consumes the 1024-dimensional multimodal embeddings produced by one of the fusion pipelines. It contains a projection network and a binary classification head. The projection network maps the input embedding through a fully connected hidden layer with 128 neurons and ReLU, then a projection layer to 64 dimensions, followed by batch normalization. A sigmoid quality head converts the projection into a scalar quality score. Training combines a custom pairwise contrastive loss with binary cross-entropy. ContrastiveNet is trained for 100 epochs using Adam with initial learning rate $10^{-3}$, batch size 64, and a 20% validation split; a scheduler halves the learning rate after 5 epochs without validation-loss improvement.

Deployment uses a threshold rule on the quality score: images with score above 0.8 are accepted for automated prediction, and those at or below 0.8 are deferred to a clinician. The paper explicitly notes that this is effectively a selective prediction policy, but it does not report risk-coverage curves, expected calibration error, conformal guarantees, uncertainty decomposition, or explicit calibration procedures beyond the thresholded quality score. A plausible implication is that the system is closer to a quality-gated screening workflow than to a formally calibrated abstention model.

5. Empirical performance

The unimodal baselines establish the performance context. An off-the-shelf pretrained ResNet18 performs poorly, with 66.4% accuracy and averaged one-vs-rest AUROC 0.534. After fine-tuning with class weighting, the image-only model rises to 88.5% accuracy and AUROC 0.986 on the held-out test set. The tabular-only neural network reaches 70.0% accuracy and AUROC 0.657. The paper argues that metadata alone are substantially weaker than imaging, but that multimodal integration improves reliability in clinically important cases (She et al., 19 Jul 2025).

For the three multimodal fusion strategies, held-out balanced test performance is reported as follows:

Fusion method	Test accuracy	Test AUROC
Concatenation	0.890	0.986
Fully connected layer fusion	0.841	0.984
Cross-attention	0.890	0.981

Average 5-fold training performance is also reported: concatenation reaches accuracy 0.929, AUROC 0.993; fully connected fusion reaches accuracy 0.881, AUROC 0.987; and cross-attention reaches accuracy 0.905, AUROC 0.988. Although fully connected fusion is not the strongest pure staging model, the paper identifies it as the most useful embedding space for the safety-oriented deferral subsystem.

A clinically salient finding concerns underestimation errors. The concatenation and cross-attention multimodal models produced zero instances of Classes 2, 3, or 4 being misclassified as Class 0 or 1, whereas the unimodal image baseline did produce such errors (She et al., 19 Jul 2025). The paper interprets this as a reduction in dangerous false negatives for more severe disease.

The deferral results are even more decisive in the paper’s narrative. When separate ContrastiveNet models are trained on embeddings from each fusion strategy, FC-fused embeddings achieve perfect held-out test accuracy in distinguishing original mBRSET images from synthesized low-quality adversarial images, while concatenation and cross-attention reach only 59.5% and 56.8% accuracy, respectively. The corresponding t-SNE visualization is reported to show strong separation between high-quality and poor-quality samples for the FC case. This is the main empirical reason the authors favor fully connected fusion overall: it produces a joint representation that is not the best classifier in isolation, but is the most effective substrate for deferral.

6. Interpretation, limitations, and relation to other “RetNet” usages

MultiRetNet’s primary significance lies in its combination of multimodal DR staging and explicit deferral. The paper’s broader claim is that integrating handheld-camera fundus images with selected metadata and routing low-quality cases to clinicians could improve early detection in underserved settings and support human-centered screening workflows (She et al., 19 Jul 2025). At the same time, several limitations are explicit or directly apparent from the experiments.

First, the multimodal fusion studies are conducted on a heavily downsampled balanced set of only 82 images per class, and the paper itself suggests that more expressive fusion schemes such as cross-attention may require much larger datasets. Second, although the motivation includes comorbidity profiles, the final multimodal model retains only four tabular features. Third, the split is random at the image level; the paper does not state that partitioning was performed by patient, even though there are multiple images per patient. Fourth, the held-out multimodal test set is class-balanced, which may not reflect deployment prevalence. Fifth, the deferral evaluation uses only 100 synthesized low-quality images, so robustness to broader out-of-distribution shifts remains unproven. Sixth, the paper reports accuracy, averaged one-vs-rest AUROC, confusion matrices, and t-SNE visualizations, but not classwise sensitivity/specificity, calibration, selective risk-coverage curves, or clinician-AI workflow outcomes. Fairness claims are similarly motivational rather than fully evaluated: the study emphasizes underserved populations and education level, but does not report subgroup metrics or fairness criteria.

The title may also invite confusion with the RetNet literature. The DR system’s image encoder is a ResNet18, not a retention-based backbone. By contrast, RetNet-related vision papers on arXiv address different problems: “Toward a Deeper Understanding: RetNet Viewed through Convolution” studies locality-biased masking for vision transformers and does not mention MultiRetNet explicitly (Li et al., 2023); “RetSeg: Retention-based Colorectal Polyps Segmentation Network” introduces a UNet-like segmentation architecture with a multi-head retention bottleneck and is likewise not called MultiRetNet (ELKarazle et al., 2023); and “SegRet: An Efficient Design for Semantic Segmentation with Retentive Network” adapts a Vision RetNet backbone to semantic segmentation without proposing a model by the name MultiRetNet (Li et al., 19 Feb 2025). This suggests that the shared “RetNet” string should not be taken as evidence of shared mechanisms. In current usage, MultiRetNet most precisely denotes a multimodal DR staging and clinician-deferral pipeline, whereas retentive-network papers concern distance-decayed token mixing, bidirectional retention, or retention-based dense prediction architectures in distinct domains.