---
title: 'SAM2-Aug: Prior Knowledge in ART Segmentation'
url: https://www.emergentmind.com/topics/sam2-aug
type: topic
---

# SAM2-Aug: Prior Knowledge in ART Segmentation

SAM2-Aug is a prior knowledge-based augmentation framework for target volume auto-segmentation in adaptive radiation therapy (ART) built on Segment Anything Model 2 (SAM2). Developed for MR-LINAC workflows, it augments prompt-based tumor segmentation with two ART-specific mechanisms: contextual incorporation of prior MR images and prior tumor annotations, and prompt-robustness enhancement through random bounding-box expansion or contraction together with mask erosion or dilation. Fine-tuned on One-Seq-Liver and evaluated without retraining on Mix-Seq-Abdomen and Mix-Seq-Brain, it reports Dice scores of 0.86 for liver, 0.89 for abdomen, and 0.90 for brain, with improvements in boundary-sensitive metrics and cross-sequence generalization [2507.19282].

## 1. Clinical setting and problem formulation

SAM2-Aug is situated in **MR-LINAC-based adaptive radiation therapy**, where accurate target delineation is required repeatedly because anatomy can change across treatment fractions. The paper frames tumor segmentation in ART as clinically critical, yet still dominated by manual contouring that is time-consuming, labor-intensive, and subject to inter-user variability. This burden is amplified in ART because repeated re-delineation is often required.

The attraction of SAM2 in this setting is its prompt-based segmentation interface. In ART, clinicians already possess strong prior information about tumor location and extent, so a promptable foundation model is a natural starting point. The paper nevertheless argues that vanilla SAM2 is insufficient for ART tumor segmentation because of a domain gap between natural-image pretraining and MR tumor imaging, the small scale of ART datasets, the variability of tumor shape and contrast, the sensitivity of SAM2 to prompt quality, and the fact that standard SAM2 does not explicitly exploit prior scans and approved prior contours that are routinely available in ART.

The targeted segmentation outputs differ by dataset. For liver and abdomen, the paper reports segmentation of **internal target volumes (ITVs)**; for brain, it reports segmentation of **clinical target volumes (CTVs)**. The method is therefore framed not as generic organ segmentation, but as target-volume delineation under longitudinal clinical constraints.

## 2. Core method and architectural use of SAM2

SAM2-Aug is organized around three components: **input data augmentation**, **prompt augmentation**, and **SAM2 fine-tuning**. It does not introduce a new standalone architecture; rather, it modifies how SAM2 is conditioned and trained for ART.

The paper summarizes the underlying promptable formulation as
\[
(m, c) = Decoder(Z^I \mid Z^P),
\]
where \(Z^I\) denotes image features extracted by the image encoder, \(Z^P\) denotes prompt features, \(m\) is the predicted binary segmentation mask, and \(c\) is the confidence score. SAM2-Aug preserves this prompt-driven logic, but changes both the image input and the prompt construction.

For adaptive fraction \(n\), the model uses the **current MRI from fraction \(n\)** together with the **prior MRI and prior tumor annotation from fraction \(n-1\)**. For the first adaptive fraction, the **simulation/planning MRI and annotation** are used as the prior. The image input is a **three-channel tensor** formed by concatenating current MRI, prior MRI, and prior tumor annotation. In parallel, the model uses both a current-slice bounding-box prompt and a prior-mask prompt. The paper writes the initial prompt set as
\[
Prompt_{init} = \{B_{cur}, M_{pri}\}, \tag{2}
\]
where \(B_{cur}\) is the current bounding box and \(M_{pri}\) is the prior mask from the prior annotation. The augmented prompt set is
\[
Prompt_{Aug} = \{A_b(B_{cur}), A_m(M_{pri})\}, \tag{3}
\]
where \(A_b\) denotes random expansion or contraction of the current bounding box and \(A_m\) denotes morphological perturbation of the prior mask.

The fine-tuning strategy is selective rather than architectural. The paper states that the **image encoder, prompt encoder, and mask decoder are fine-tuned jointly via backpropagation**, and its ablation identifies this combination as the strongest-performing configuration among the tested module subsets [2507.19282].

## 3. Prior-knowledge injection and prompt-robustness strategy

A defining feature of SAM2-Aug is that **prior ART information is injected as image channels rather than as memory-bank entries or pseudo-labels**. The current MRI, prior MRI, and prior annotation are concatenated directly into the model input. Before this, the current MRI is spatially aligned with the prior scan using **rigid registration** in the prompt generation stage. This makes prior anatomy and prior tumor geometry available to the image encoder as contextual evidence.

The second defining feature is prompt augmentation. During training, the current bounding box is perturbed by **random expansions or contractions of 0–5 pixels in all directions**, while the prior mask is perturbed by **erosion or dilation**. The paper also applies a **random selection mechanism** to the augmented box and mask prompts, and describes the strategy as involving **randomly dropping out bounding boxes or masks** to improve robustness to incomplete prompt availability.

At test time, the behavior changes. The paper states that bounding boxes are **similarly expanded or contracted by 0 to 5 pixels in each direction, for a randomly selected subset of the four directions (left, right, top, bottom)**. By contrast, the **prior tumor annotation is not augmented**, and **no dropout is performed**, because physician-approved prior information is assumed to be available in ART. A separate robustness experiment perturbs bounding boxes by **1–10 pixels** in **one to four directions**, and the paper reports that the augmentation strategy yields more stable mean Dice under such prompt variation.

This design makes the prompt interface explicitly stochastic during training but not during deployment. A plausible implication is that SAM2-Aug uses prompt noise as regularization while treating prior clinical contours as reliable conditioning at inference time.

## 4. Datasets, training configuration, and evaluation protocol

The study uses three MRI datasets derived from ART workflows. One-Seq-Liver is the only dataset used for model development; the abdomen and brain datasets are reserved for direct generalization tests without retraining.

| Dataset | Composition | Role |
|---|---|---|
| One-Seq-Liver | 31 liver cancer patients, 115 MRI scans, 167 ITVs | Training, validation, testing |
| Mix-Seq-Abdomen | 28 patients, 88 fractional MRI scans, 156 ITVs | External test only |
| Mix-Seq-Brain | 37 patients, 86 MRI scans, 97 CTVs | External test only |

One-Seq-Liver consists of **T2-weighted MultiVane extended Dynamic (T2-MVXD)** scans from a **1.5T MR-LINAC (Unity, Elekta)**, with **1–7 fractions per patient** and mean target volume **45,021 mm\(^3\)**. Mix-Seq-Abdomen includes abdominal cancers such as liver, pancreatic, and renal tumors, with sequences **T2-MVXD**, **T2-MVXD-SPIR**, **T1-3D-VaneXD**, and **T2-FS**, and mean target volume **26,664 mm\(^3\)**. Mix-Seq-Brain includes **T2-FLAIR**, **T1**, and **T1 with contrast** scans, with mean target volume **156,772 mm\(^3\)**. Simulation scans exist for all datasets but are not counted in scan totals because they are used only as priors for the first adaptive fraction.

For One-Seq-Liver, the split is **19 patients and 69 scans for training**, **5 patients and 17 scans for validation**, and **7 patients and 29 scans for testing**. The implementation uses **pretrained SAM2-Tiny**, **batch size 4**, **AdamW**, initial learning rate **\(6 \times 10^{-6}\)**, weight decay **0.01**, and **5 epochs** on a **single NVIDIA RTX 4090**. The paper states that training uses a **combination of Dice loss and cross-entropy loss**, and that more epochs did not improve validation performance.

Evaluation uses **Dice**, **Normalized Surface Dice (NSD)**, **HD95**, and **ASD**. The paper gives
\[
NSD =  \frac{ |\{x \in S_G: d(x, S_P) < \tau\}| + |\{x \in S_P: d(x, S_G) < \tau\}| }{ |S_G| + |S_P| }, \tag{5}
\]
with \(\tau = 0.5\), and defines
\[
HD95(S_G, S_P) = \max \left( P_{a \in S_G}\min_{b \in S_P} d(a,b), P_{b \in S_P}\min_{a \in S_G} d(b,a) \right), \tag{6}
\]
and
\[
ASD(S_G, S_P) = \frac{1}{|S_G| + |S_P|} \left( \sum_{x \in S_G} d(x, S_P) + \sum_{x \in S_P} d(x, S_G) \right). \tag{7}
\]
The paper also prints a Dice equation, but notes in the provided summary that the denominator appears malformed in the OCR rendering [2507.19282].

## 5. Quantitative performance and ablation evidence

The main empirical claim is that SAM2-Aug outperforms conventional CNNs, transformer-based medical segmentation models, and several SAM-family baselines across all three datasets.

| Dataset | SAM2-Aug Dice | Vanilla SAM2 Dice |
|---|---:|---:|
| One-Seq-Liver | **0.86 ± 0.08** | 0.78 ± 0.13 |
| Mix-Seq-Abdomen | **0.89 ± 0.06** | 0.84 ± 0.11 |
| Mix-Seq-Brain | **0.90 ± 0.07** | 0.88 ± 0.08 |

On **One-Seq-Liver**, SAM2-Aug reaches **Dice 0.86 ± 0.08**, **NSD 0.58 ± 0.12**, **HD95 1.98 ± 0.66**, and **ASD 0.60 ± 0.23**. This exceeds **registration-only** at **0.83 ± 0.13 Dice**, **SegResNet** at **0.82 ± 0.11**, **MedSAM** at **0.83 ± 0.09**, and vanilla **SAM2** at **0.78 ± 0.13**. On **Mix-Seq-Abdomen**, tested without retraining, SAM2-Aug reaches **0.89 ± 0.06 Dice** against **0.84 ± 0.11** for SAM2, **0.84 ± 0.08** for MedSAM, and **0.83 ± 0.14** for SegResNet. On **Mix-Seq-Brain**, it reaches **0.90 ± 0.07 Dice**, compared with **0.88 ± 0.08** for SAM2 and **0.84 ± 0.09** for SAM-Med2D. The boundary-sensitive gains on brain are especially large: **HD95 3.96 vs 9.86** and **ASD 1.15 vs 2.34** relative to vanilla SAM2.

The ablations isolate both augmentation mechanisms. For **input data augmentation** on One-Seq-Liver, using only **curMR** yields **0.75 ± 0.12 Dice**; adding **priMR** gives **0.80 ± 0.12**; adding **priSeg** gives **0.82 ± 0.10**; and using **curMR + priMR + priSeg** yields the full **0.86 ± 0.08**. This indicates that both prior imaging and prior contour information contribute, with the full combination adding **0.11 Dice** over current MRI alone.

For **prompt ablation**, the paper reports **Mask 0.73 ± 0.25**, **BBox 0.82 ± 0.10**, **AugBBox 0.84 ± 0.08**, **BBox + Mask 0.76 ± 0.12**, and **AugBBox + AugMask 0.86 ± 0.08**. The authors summarize four patterns: bounding boxes outperform mask-only prompts; augmenting bounding boxes improves Dice by **0.02**; combining box and mask without augmentation is worse than box alone; and the best performance is obtained only when **both** prompt types are augmented.

For **module fine-tuning**, the paper compares **MskDec**, **ImgEnc**, **ImgEnc + MskDec**, **ImgEnc + MskDec + PrmpEnc**, and **All modules**. The best result is achieved by **ImgEnc + MskDec + PrmpEnc**, with **31.4M** parameters and **Dice 0.86 ± 0.08**. Fine-tuning **all modules** reaches **0.85 ± 0.09**, slightly below the best selective configuration. This suggests that targeted adaptation of the image encoder, prompt encoder, and mask decoder is more effective than indiscriminate full-module updating.

The robustness experiment further reports that the proposed prompt augmentation yields **more stable mean Dice** under random box expansion of **1–10 pixels** in **one to four directions** on both One-Seq-Liver and Mix-Seq-Abdomen. The paper also notes that mask prompts were especially helpful for **small tumors** [2507.19282].

## 6. Position within SAM2 augmentation research and reported limitations

SAM2-Aug belongs to a broader class of methods that adapt SAM2 to domain-specific segmentation tasks, but its mechanism is unusually tied to the ART workflow. The SAM2 survey literature characterizes the wider field in terms of adaptation, prompt design, robustness, and domain transfer [2503.12781]. Within that landscape, SAM2-Aug is most closely aligned with **input/prompt augmentation and selective fine-tuning**, rather than with architectural replacement of the backbone or memory-control redesign.

Other SAM2 augmentation lines emphasize different intervention points. **MAE-SAM2** augments the **image encoder** through self-supervised MAE pretraining and a task-specific Dice+BCE loss for retinal vascular leakage segmentation [2509.10554]. **CRISP-SAM2** replaces geometric prompts with **semantic prompting**, injects cross-modal image-text semantics into the frozen encoder, adds a local refiner, and modifies the memory update strategy for 3D multi-organ CT [2506.23121]. **UniUltra** uses a **context-edge hybrid adapter** and a distillation pipeline for ultrasound, achieving parameter-efficient fine-tuning with **8.91%** of SAM2 parameters during adaptation [2511.15771]. By contrast, **SAM2RL** and **MA-SAM2** target the **memory policy** of SAM2 rather than its image input or prompt representation: the former learns memory updates with reinforcement learning [2507.08548], while the latter uses context-aware and occlusion-resilient heuristic memories for surgical video segmentation [2507.09577]. This suggests that SAM2-Aug occupies a distinct subfamily centered on **longitudinal prior knowledge and prompt robustness** rather than multimodal prompting or memory selection.

The paper also states several limitations. It reports that **one/few prompts during inference did not work well** for tumor segmentation in ART, despite SAM2’s video-style memory mechanism. It also acknowledges the lack of **explicit 3D modeling**, noting that the current implementation is effectively slice-wise and that **3D adapters** are a future direction. The scope of prompt augmentation is limited to **box and mask prompts**, with future work proposed on **3D prompts, points, scribbles, and lasso prompts**. The summary also notes that some implementation details remain unspecified, including morphological kernel settings for erosion or dilation, probabilities for prompt dropout, the exact test-time prompt realization strategy, and image normalization or resampling details. Within the paper’s own framing, SAM2-Aug is therefore best understood as a clinically grounded and empirically strong ART-specific adaptation of SAM2, but not yet as a full volumetric or universally specified medical segmentation framework [2507.19282].

Source: https://www.emergentmind.com/topics/sam2-aug