SAM2-Aug: Prior Knowledge in ART Segmentation

• SAM2-Aug is a prior knowledge-based augmentation framework that integrates current and prior MR images and annotations for improved tumor segmentation in ART.
• It employs input and prompt augmentations, using random bounding-box perturbations and mask erosion/dilation to enhance segmentation robustness.
• Evaluated on liver, abdomen, and brain datasets, SAM2-Aug achieves higher Dice scores and superior boundary metrics compared to baseline SAM2 and other methods.

SAM2-Aug is a prior knowledge-based augmentation framework for target volume auto-segmentation in adaptive radiation therapy (ART) built on Segment Anything Model 2 (SAM2). Developed for MR-LINAC workflows, it augments prompt-based tumor segmentation with two ART-specific mechanisms: contextual incorporation of prior MR images and prior tumor annotations, and prompt-robustness enhancement through random bounding-box expansion or contraction together with mask erosion or dilation. Fine-tuned on One-Seq-Liver and evaluated without retraining on Mix-Seq-Abdomen and Mix-Seq-Brain, it reports Dice scores of 0.86 for liver, 0.89 for abdomen, and 0.90 for brain, with improvements in boundary-sensitive metrics and cross-sequence generalization (Xu et al., 25 Jul 2025).

1. Clinical setting and problem formulation

SAM2-Aug is situated in MR-LINAC-based adaptive radiation therapy, where accurate target delineation is required repeatedly because anatomy can change across treatment fractions. The paper frames tumor segmentation in ART as clinically critical, yet still dominated by manual contouring that is time-consuming, labor-intensive, and subject to inter-user variability. This burden is amplified in ART because repeated re-delineation is often required.

The attraction of SAM2 in this setting is its prompt-based segmentation interface. In ART, clinicians already possess strong prior information about tumor location and extent, so a promptable foundation model is a natural starting point. The paper nevertheless argues that vanilla SAM2 is insufficient for ART tumor segmentation because of a domain gap between natural-image pretraining and MR tumor imaging, the small scale of ART datasets, the variability of tumor shape and contrast, the sensitivity of SAM2 to prompt quality, and the fact that standard SAM2 does not explicitly exploit prior scans and approved prior contours that are routinely available in ART.

The targeted segmentation outputs differ by dataset. For liver and abdomen, the paper reports segmentation of internal target volumes (ITVs); for brain, it reports segmentation of clinical target volumes (CTVs). The method is therefore framed not as generic organ segmentation, but as target-volume delineation under longitudinal clinical constraints.

2. Core method and architectural use of SAM2

SAM2-Aug is organized around three components: input data augmentation, prompt augmentation, and SAM2 fine-tuning. It does not introduce a new standalone architecture; rather, it modifies how SAM2 is conditioned and trained for ART.

The paper summarizes the underlying promptable formulation as

$(m, c) = Decoder(Z^I \mid Z^P),$

where $Z^I$ denotes image features extracted by the image encoder, $Z^P$ denotes prompt features, $m$ is the predicted binary segmentation mask, and $c$ is the confidence score. SAM2-Aug preserves this prompt-driven logic, but changes both the image input and the prompt construction.

For adaptive fraction $n$, the model uses the current MRI from fraction $n$ together with the prior MRI and prior tumor annotation from fraction $n-1$. For the first adaptive fraction, the simulation/planning MRI and annotation are used as the prior. The image input is a three-channel tensor formed by concatenating current MRI, prior MRI, and prior tumor annotation. In parallel, the model uses both a current-slice bounding-box prompt and a prior-mask prompt. The paper writes the initial prompt set as

$Prompt_{init} = \{B_{cur}, M_{pri}\}, \tag{2}$

where $B_{cur}$ is the current bounding box and $Z^I$0 is the prior mask from the prior annotation. The augmented prompt set is

$Z^I$1

where $Z^I$2 denotes random expansion or contraction of the current bounding box and $Z^I$3 denotes morphological perturbation of the prior mask.

The fine-tuning strategy is selective rather than architectural. The paper states that the image encoder, prompt encoder, and mask decoder are fine-tuned jointly via backpropagation, and its ablation identifies this combination as the strongest-performing configuration among the tested module subsets (Xu et al., 25 Jul 2025).

3. Prior-knowledge injection and prompt-robustness strategy

A defining feature of SAM2-Aug is that prior ART information is injected as image channels rather than as memory-bank entries or pseudo-labels. The current MRI, prior MRI, and prior annotation are concatenated directly into the model input. Before this, the current MRI is spatially aligned with the prior scan using rigid registration in the prompt generation stage. This makes prior anatomy and prior tumor geometry available to the image encoder as contextual evidence.

The second defining feature is prompt augmentation. During training, the current bounding box is perturbed by random expansions or contractions of 0–5 pixels in all directions, while the prior mask is perturbed by erosion or dilation. The paper also applies a random selection mechanism to the augmented box and mask prompts, and describes the strategy as involving randomly dropping out bounding boxes or masks to improve robustness to incomplete prompt availability.

At test time, the behavior changes. The paper states that bounding boxes are similarly expanded or contracted by 0 to 5 pixels in each direction, for a randomly selected subset of the four directions (left, right, top, bottom). By contrast, the prior tumor annotation is not augmented, and no dropout is performed, because physician-approved prior information is assumed to be available in ART. A separate robustness experiment perturbs bounding boxes by 1–10 pixels in one to four directions, and the paper reports that the augmentation strategy yields more stable mean Dice under such prompt variation.

This design makes the prompt interface explicitly stochastic during training but not during deployment. A plausible implication is that SAM2-Aug uses prompt noise as regularization while treating prior clinical contours as reliable conditioning at inference time.

4. Datasets, training configuration, and evaluation protocol

The study uses three MRI datasets derived from ART workflows. One-Seq-Liver is the only dataset used for model development; the abdomen and brain datasets are reserved for direct generalization tests without retraining.

Dataset	Composition	Role
One-Seq-Liver	31 liver cancer patients, 115 MRI scans, 167 ITVs	Training, validation, testing
Mix-Seq-Abdomen	28 patients, 88 fractional MRI scans, 156 ITVs	External test only
Mix-Seq-Brain	37 patients, 86 MRI scans, 97 CTVs	External test only

One-Seq-Liver consists of T2-weighted MultiVane extended Dynamic (T2-MVXD) scans from a 1.5T MR-LINAC (Unity, Elekta), with 1–7 fractions per patient and mean target volume 45,021 mm$Z^I$4. Mix-Seq-Abdomen includes abdominal cancers such as liver, pancreatic, and renal tumors, with sequences T2-MVXD, T2-MVXD-SPIR, T1-3D-VaneXD, and T2-FS, and mean target volume 26,664 mm$Z^I$5. Mix-Seq-Brain includes T2-FLAIR, T1, and T1 with contrast scans, with mean target volume 156,772 mm$Z^I$6. Simulation scans exist for all datasets but are not counted in scan totals because they are used only as priors for the first adaptive fraction.

For One-Seq-Liver, the split is 19 patients and 69 scans for training, 5 patients and 17 scans for validation, and 7 patients and 29 scans for testing. The implementation uses pretrained SAM2-Tiny, batch size 4, AdamW, initial learning rate $Z^I$7, weight decay 0.01, and 5 epochs on a single NVIDIA RTX 4090. The paper states that training uses a combination of Dice loss and cross-entropy loss, and that more epochs did not improve validation performance.

Evaluation uses Dice, Normalized Surface Dice (NSD), HD95, and ASD. The paper gives

$Z^I$8

with $Z^I$9, and defines

$Z^P$0

and

$Z^P$1

The paper also prints a Dice equation, but notes in the provided summary that the denominator appears malformed in the OCR rendering (Xu et al., 25 Jul 2025).

5. Quantitative performance and ablation evidence

The main empirical claim is that SAM2-Aug outperforms conventional CNNs, transformer-based medical segmentation models, and several SAM-family baselines across all three datasets.

Dataset	SAM2-Aug Dice	Vanilla SAM2 Dice
One-Seq-Liver	0.86 ± 0.08	0.78 ± 0.13
Mix-Seq-Abdomen	0.89 ± 0.06	0.84 ± 0.11
Mix-Seq-Brain	0.90 ± 0.07	0.88 ± 0.08

On One-Seq-Liver, SAM2-Aug reaches Dice 0.86 ± 0.08, NSD 0.58 ± 0.12, HD95 1.98 ± 0.66, and ASD 0.60 ± 0.23. This exceeds registration-only at 0.83 ± 0.13 Dice, SegResNet at 0.82 ± 0.11, MedSAM at 0.83 ± 0.09, and vanilla SAM2 at 0.78 ± 0.13. On Mix-Seq-Abdomen, tested without retraining, SAM2-Aug reaches 0.89 ± 0.06 Dice against 0.84 ± 0.11 for SAM2, 0.84 ± 0.08 for MedSAM, and 0.83 ± 0.14 for SegResNet. On Mix-Seq-Brain, it reaches 0.90 ± 0.07 Dice, compared with 0.88 ± 0.08 for SAM2 and 0.84 ± 0.09 for SAM-Med2D. The boundary-sensitive gains on brain are especially large: HD95 3.96 vs 9.86 and ASD 1.15 vs 2.34 relative to vanilla SAM2.

The ablations isolate both augmentation mechanisms. For input data augmentation on One-Seq-Liver, using only curMR yields 0.75 ± 0.12 Dice; adding priMR gives 0.80 ± 0.12; adding priSeg gives 0.82 ± 0.10; and using curMR + priMR + priSeg yields the full 0.86 ± 0.08. This indicates that both prior imaging and prior contour information contribute, with the full combination adding 0.11 Dice over current MRI alone.

For prompt ablation, the paper reports Mask 0.73 ± 0.25, BBox 0.82 ± 0.10, AugBBox 0.84 ± 0.08, BBox + Mask 0.76 ± 0.12, and AugBBox + AugMask 0.86 ± 0.08. The authors summarize four patterns: bounding boxes outperform mask-only prompts; augmenting bounding boxes improves Dice by 0.02; combining box and mask without augmentation is worse than box alone; and the best performance is obtained only when both prompt types are augmented.

For module fine-tuning, the paper compares MskDec, ImgEnc, ImgEnc + MskDec, ImgEnc + MskDec + PrmpEnc, and All modules. The best result is achieved by ImgEnc + MskDec + PrmpEnc, with 31.4M parameters and Dice 0.86 ± 0.08. Fine-tuning all modules reaches 0.85 ± 0.09, slightly below the best selective configuration. This suggests that targeted adaptation of the image encoder, prompt encoder, and mask decoder is more effective than indiscriminate full-module updating.

The robustness experiment further reports that the proposed prompt augmentation yields more stable mean Dice under random box expansion of 1–10 pixels in one to four directions on both One-Seq-Liver and Mix-Seq-Abdomen. The paper also notes that mask prompts were especially helpful for small tumors (Xu et al., 25 Jul 2025).

6. Position within SAM2 augmentation research and reported limitations

SAM2-Aug belongs to a broader class of methods that adapt SAM2 to domain-specific segmentation tasks, but its mechanism is unusually tied to the ART workflow. The SAM2 survey literature characterizes the wider field in terms of adaptation, prompt design, robustness, and domain transfer (Jiaxing et al., 17 Mar 2025). Within that landscape, SAM2-Aug is most closely aligned with input/prompt augmentation and selective fine-tuning, rather than with architectural replacement of the backbone or memory-control redesign.

Other SAM2 augmentation lines emphasize different intervention points. MAE-SAM2 augments the image encoder through self-supervised MAE pretraining and a task-specific Dice+BCE loss for retinal vascular leakage segmentation (Xing et al., 9 Sep 2025). CRISP-SAM2 replaces geometric prompts with semantic prompting, injects cross-modal image-text semantics into the frozen encoder, adds a local refiner, and modifies the memory update strategy for 3D multi-organ CT (Yu et al., 29 Jun 2025). UniUltra uses a context-edge hybrid adapter and a distillation pipeline for ultrasound, achieving parameter-efficient fine-tuning with 8.91% of SAM2 parameters during adaptation (Li et al., 19 Nov 2025). By contrast, SAM2RL and MA-SAM2 target the memory policy of SAM2 rather than its image input or prompt representation: the former learns memory updates with reinforcement learning (Adamyan et al., 11 Jul 2025), while the latter uses context-aware and occlusion-resilient heuristic memories for surgical video segmentation (Yin et al., 13 Jul 2025). This suggests that SAM2-Aug occupies a distinct subfamily centered on longitudinal prior knowledge and prompt robustness rather than multimodal prompting or memory selection.

The paper also states several limitations. It reports that one/few prompts during inference did not work well for tumor segmentation in ART, despite SAM2’s video-style memory mechanism. It also acknowledges the lack of explicit 3D modeling, noting that the current implementation is effectively slice-wise and that 3D adapters are a future direction. The scope of prompt augmentation is limited to box and mask prompts, with future work proposed on 3D prompts, points, scribbles, and lasso prompts. The summary also notes that some implementation details remain unspecified, including morphological kernel settings for erosion or dilation, probabilities for prompt dropout, the exact test-time prompt realization strategy, and image normalization or resampling details. Within the paper’s own framing, SAM2-Aug is therefore best understood as a clinically grounded and empirically strong ART-specific adaptation of SAM2, but not yet as a full volumetric or universally specified medical segmentation framework (Xu et al., 25 Jul 2025).