MAE-SAM2: Retinal Leakage Segmentation

• The paper demonstrates that MAE pretraining of the SAM2 encoder yields a 5.8% relative Dice improvement, enhancing segmentation of small, densely distributed leakage regions.
• The method uses a two-stage pipeline: first, MAE pretraining on unlabeled FA images, then fine-tuning on annotated leakage masks, effectively combining Dice and BCE losses.
• MAE-SAM2 is clinically significant by reducing annotation requirements and improving reliability in quantifying retinal vascular leakage even with imaging artifacts.

MAE-SAM2 is a foundation model for retinal vascular leakage segmentation on fluorescein angiography (FA) images that combines Segment Anything Model v2 (SAM2) with Masked Autoencoder (MAE) self-supervised pretraining of the SAM2 image encoder. It is designed for a regime in which leakage areas are small and densely distributed, while labeled clinical data are limited. The method uses a two-stage pipeline—MAE pretraining on unlabeled FA images followed by task-specific fine-tuning on leakage masks—and, on the reported test set, achieves the highest Dice score and Intersection-over-Union (IoU) among the compared models, including a relative Dice improvement of approximately $5.8\%$ over the original SAM2 (Xing et al., 9 Sep 2025).

1. Problem setting and model scope

MAE-SAM2 targets binary segmentation of retinal vascular leakage in FA imaging. In the formulation reported for the method, an input image is $x \in \mathbb{R}^{H \times W \times 3}$, the binary ground-truth leakage mask is $y \in \{0,1\}^{H \times W}$, and the final prediction is written as

$$\hat y=\mathrm{SAM2}_{\theta^{*},\psi}(x;\text{prompt}),$$

where $\theta^{*}$ denotes the MAE-pretrained encoder weights and $\psi$ denotes the mask-decoder weights (Xing et al., 9 Sep 2025).

The principal motivation is task difficulty arising from three conditions stated explicitly for the application: the leakage regions are small, their distribution is dense, and labeled clinical data are scarce. Within that setting, the paper positions self-supervised pretraining not as a generic regularizer but as a mechanism for learning robust visual representations from unlabeled FA images before supervised adaptation to leakage masks. This suggests that the method is specifically tuned to the representation bottlenecks of high-resolution ophthalmic angiography rather than merely transplanting a generic segmentation backbone.

The model is also presented as a foundation-model adaptation rather than a de novo segmentation architecture. Its significance therefore lies in how SAM2 is specialized for a narrow clinical target: the image encoder is altered through MAE pretraining, while the downstream SAM2 segmentation machinery is retained structurally.

2. Architectural composition and two-stage training

MAE-SAM2 is built from two components: SAM2 as the backbone segmentation foundation model, and MAE self-supervised pretraining applied to the SAM2 image encoder. Training proceeds in two stages. Stage 1 performs MAE pretraining on unlabeled FA images. Stage 2 fine-tunes the full SAM2—encoder and decoder—on leakage masks (Xing et al., 9 Sep 2025).

The SAM2 decomposition used in the method is summarized as follows. The image encoder is a ViT transformer stack that produces patch embeddings. The prompt encoder embeds user prompts, including point, box, mask, or none. The memory module, described for video, stores key representations for temporal consistency. The mask decoder is a lightweight ViT decoder that uses image and prompt features to predict a mask. In MAE-SAM2, the pretraining intervention is restricted to the image encoder, but the fine-tuning stage updates the image encoder together with the prompt encoder and mask decoder weights, while leaving their structures unchanged.

During MAE pretraining, a high proportion of input patches is randomly masked with mask ratio $r$. The masked input is passed through the SAM2 image encoder $E_{\theta}$,

$z = E_{\theta}(\mathrm{mask}(x)),$

and a lightweight convolutional decoder $D_{\phi}$ reconstructs the original image,

$x \in \mathbb{R}^{H \times W \times 3}$0

The MAE decoder is used only during pretraining; its weights $x \in \mathbb{R}^{H \times W \times 3}$1 are discarded after Stage 1. The pretrained encoder weights $x \in \mathbb{R}^{H \times W \times 3}$2 then initialize the SAM2 encoder in Stage 2 (Xing et al., 9 Sep 2025).

This encoder-centric modification is central to the method’s identity. MAE-SAM2 does not redefine SAM2’s prompt or mask-decoding interfaces; instead, it changes the initialization of the visual representation space before clinical fine-tuning. A plausible implication is that the method seeks to preserve as much of SAM2’s pretrained structural prior as possible while adapting only the encoder statistics to FA imagery.

3. Objectives and optimization strategy

The Stage 1 objective is the MAE reconstruction loss over masked patches: $x \in \mathbb{R}^{H \times W \times 3}$3 where $x \in \mathbb{R}^{H \times W \times 3}$4 is the set of masked patch indices. This is the sole active objective in pretraining, implemented by setting $x \in \mathbb{R}^{H \times W \times 3}$5 and $x \in \mathbb{R}^{H \times W \times 3}$6 in the total loss (Xing et al., 9 Sep 2025).

For segmentation, the reported candidate losses include Dice loss,

$x \in \mathbb{R}^{H \times W \times 3}$7

IoU loss,

$x \in \mathbb{R}^{H \times W \times 3}$8

and binary cross-entropy (BCE),

$x \in \mathbb{R}^{H \times W \times 3}$9

with $y \in \{0,1\}^{H \times W}$0 and $y \in \{0,1\}^{H \times W}$1.

The task-specific combined loss ultimately selected for fine-tuning is

$y \in \{0,1\}^{H \times W}$2

In Stage 2, the reported setting is $y \in \{0,1\}^{H \times W}$3, $y \in \{0,1\}^{H \times W}$4, $y \in \{0,1\}^{H \times W}$5, and $y \in \{0,1\}^{H \times W}$6 (Xing et al., 9 Sep 2025).

The paper’s ablation indicates that the combined Dice+BCE loss contributes only a modest gain when used alone, whereas MAE pretraining has a larger standalone effect. This suggests that, for this task, representation quality learned from unlabeled FA data is more consequential than loss engineering in isolation.

4. Dataset, preprocessing, and implementation details

The dataset consists of 74 FA images from 38 patients, each with original resolution $y \in \{0,1\}^{H \times W}$7, together with expert-drawn binary masks for vascular leakage. The train/test split is performed at subject level with an 80/20 partition: 53 scans from 28 patients for training, and 21 scans from 10 patients for testing (Xing et al., 9 Sep 2025).

Training uses $y \in \{0,1\}^{H \times W}$8 patches extracted with stride 512, expanding the training set to approximately 2000 patches. Validation and test images are resized as whole images to $y \in \{0,1\}^{H \times W}$9. Standard augmentations are applied: horizontal and vertical flip, small rotations of $$\hat y=\mathrm{SAM2}_{\theta^{*},\psi}(x;\text{prompt}),$$0, and brightness/contrast jitter.

The optimization schedule is also fully specified. MAE pretraining is run for 400 epochs with mask ratio $$\hat y=\mathrm{SAM2}_{\theta^{*},\psi}(x;\text{prompt}),$$1, AdamW, learning rate $$\hat y=\mathrm{SAM2}_{\theta^{*},\psi}(x;\text{prompt}),$$2, batch size 16, and weight decay 0.05. Fine-tuning is run for 50 epochs with the same optimizer settings and uses learning-rate warmup plus LambdaLR decay. The reported hardware is $$\hat y=\mathrm{SAM2}_{\theta^{*},\psi}(x;\text{prompt}),$$3 NVIDIA A100 GPUs with mixed precision (Xing et al., 9 Sep 2025).

These details are important for interpreting the results. The training regime couples high-resolution retinal images with aggressive patch extraction, which is consistent with the stated challenge of small and diffuse leakage regions. The subject-level split also matters because it avoids a patient-overlap confound between train and test partitions.

5. Quantitative performance and ablation evidence

On the reported test set, MAE-SAM2 outperforms the compared models in Dice and IoU, and also attains the best precision, recall, $$\hat y=\mathrm{SAM2}_{\theta^{*},\psi}(x;\text{prompt}),$$4, and specificity values among the listed methods (Xing et al., 9 Sep 2025).

Model	Dice	IoU
U-Net	0.2426	0.1481
U-Net++	0.2668	0.1655
DeepLabV3+	0.3502	0.2647
Swin-Unet	0.2543	0.1573
SAM2	0.5288	0.4092
MAE-SAM2	0.5593	0.4348

For MAE-SAM2 specifically, the full reported metric vector is Dice $$\hat y=\mathrm{SAM2}_{\theta^{*},\psi}(x;\text{prompt}),$$5, IoU $$\hat y=\mathrm{SAM2}_{\theta^{*},\psi}(x;\text{prompt}),$$6, precision $$\hat y=\mathrm{SAM2}_{\theta^{*},\psi}(x;\text{prompt}),$$7, recall $$\hat y=\mathrm{SAM2}_{\theta^{*},\psi}(x;\text{prompt}),$$8, $$\hat y=\mathrm{SAM2}_{\theta^{*},\psi}(x;\text{prompt}),$$9 $\theta^{*}$0, and specificity $\theta^{*}$1. For SAM2, the corresponding values are Dice $\theta^{*}$2, IoU $\theta^{*}$3, precision $\theta^{*}$4, recall $\theta^{*}$5, $\theta^{*}$6 $\theta^{*}$7, and specificity $\theta^{*}$8. The paper computes the Dice improvement as

$\theta^{*}$9

and reports an IoU gain of $\psi$0 relative to SAM2.

The ablation study isolates the contributions of MAE pretraining and the combined loss:

MAE pretraining	Combined Dice+BCE loss	Dice
–	–	0.5288
–	✓	0.5305
✓	–	0.5438
✓	✓	0.5593

The interpretation given is that MAE pretraining alone yields a $\psi$1 absolute Dice gain, the combined Dice+BCE loss alone yields $\psi$2, and both together are synergistic (Xing et al., 9 Sep 2025). The factual pattern in the table supports the claim that encoder pretraining is the dominant contributor, while the combined loss refines the already-improved model.

6. Clinical significance, limitations, and terminological distinctions

The paper links the segmentation improvements to clinical relevance in three ways. First, improved Dice and IoU for small, diffuse vascular leakage are described as enabling more reliable, repeatable quantification of disease activity in retinal vasculitis. Second, MAE-SAM2 is reported to be robust to imaging artifacts such as overexposure, thereby reducing false positives in healthy regions. Third, the self-supervised pretraining stage is presented as lowering annotation requirements and thus making the approach practical in data-scarce clinical settings (Xing et al., 9 Sep 2025).

The stated limitations and future directions are also specific. The current model uses point prompts; exploration of mask prompts or fully zero-shot inference remains open. Efficient fine-tuning methods such as adapters or LoRA are proposed as possible ways to reduce compute. External validation on multi-center FA datasets is identified as necessary to confirm generalization. These points delimit the current evidence base: the model is not yet presented as externally validated across institutions, nor as prompt-free in the strong sense of fully zero-shot deployment.

A common source of confusion is nomenclature. MAE-SAM2 should be distinguished from MA-SAM2, the “Memory-Augmented SAM2” method for training-free surgical video segmentation. MA-SAM2 introduces context-aware memory and occlusion-resilient memory branches, uses multi-target, single-loop, one-prompt inference, adds no parameters, and reports gains over SAM2 on EndoVis2017 and EndoVis2018. By contrast, MAE-SAM2 is a retinal FA image-segmentation method centered on MAE self-supervised pretraining of the SAM2 image encoder and supervised fine-tuning on leakage masks (Yin et al., 13 Jul 2025). The similarity of acronyms masks a substantial methodological difference: one approach modifies representation learning through pretraining for clinical retinal images, while the other modifies memory management for training-free video segmentation.