---
title: 'ST-SAM: Self-Training with SAM Refinement'
url: https://www.emergentmind.com/topics/st-sam
type: topic
---

# ST-SAM: Self-Training with SAM Refinement

ST-SAM denotes a self-training formulation that integrates the Segment Anything Model (SAM) into semi-supervised dense prediction by using model-generated pseudo-labels as the basis for SAM-guided refinement. In the 2025 literature, the name appears explicitly in “ST-SAM: SAM-Driven Self-Training Framework for Semi-Supervised Camouflaged Object Detection” and equivalently in remote sensing semantic segmentation as “ST-SAM/SAMST (Self-Training with SAM),” where a supervised segmentation model alternates with a SAM-based Pseudo-label Refiner [2507.23307; 2507.11994]. In both usages, the central premise is the same: dense pixel annotation is scarce, unlabeled imagery is abundant, and naive self-training is limited by noisy pseudo-labels and weak boundary quality. ST-SAM addresses this by coupling confidence-based pseudo-label curation with SAM’s zero-shot mask generation.

## 1. Terminology, problem setting, and naming overlap

The term **ST-SAM** is not confined to a single task or a single fixed architecture. In semi-supervised camouflaged object detection, ST-SAM is introduced as a **SAM-Driven Self-Training Framework for Semi-Supervised Camouflaged Object Detection**. In remote sensing semantic segmentation, the same idea is described as **ST-SAM/SAMST (Self-Training with SAM)**, with “SAMST” emphasizing SAM’s role in self-training and “ST-SAM” denoting the same integration pattern [2507.23307; 2507.11994].

The two problem domains differ materially. In semi-supervised camouflaged object detection, the difficulty arises from **large scale variations, complex contours, and textures that are often indistinguishable from the background**, making high-quality pixel-level annotations expensive and scarce. In remote sensing semantic segmentation, the motivating factors are **label scarcity**, **resolution variability**, and **inconsistent land-cover definitions** across sources, all of which hinder direct supervised scaling. Despite these differences, both formulations are organized around the same failure mode of standard pseudo-labeling: early pseudo-labels are informative enough to expand supervision, but sufficiently noisy to contaminate later optimization if they are used without filtration and geometric correction.

A common misconception is that ST-SAM names a single architectural template. The published descriptions indicate the opposite. The camouflaged object detection variant is explicitly **model-agnostic** and is demonstrated with a standard encoder–decoder using a ConvNeXt-B backbone, while the remote sensing variant uses a **Swin Transformer** with **UperHead** and an auxiliary **FCNHead**. What is shared is not a backbone, but a training logic in which pseudo-labels are first filtered, then converted into SAM prompts, and finally reinjected into model training after SAM-assisted correction.

## 2. ST-SAM for semi-supervised camouflaged object detection

In semi-supervised camouflaged object detection, the labeled and unlabeled sets are denoted
$D_l = \{(x_i^l, y_i)\}_{i=1}^M$ and $D_u = \{x_j^u\}_{j=1}^N$ with $M \ll N$, and the single segmentation model is written as $f_\theta : x \rightarrow \mathrm{Pred}$, where $\mathrm{Pred}$ is a soft foreground probability map. The initial pseudo-label for an unlabeled image is $P^I = f_\theta(x)$ [2507.23307].

The framework is motivated by explicit limitations of teacher–student SSCOD. Under very scarce supervision, multi-network pipelines are described as suffering from **prediction bias**, **error propagation**, and increased **computational overhead and scalability** constraints. ST-SAM replaces that regime with **single-model self-training**, with the stated goal of fundamentally circumventing inter-model prediction bias while maintaining annotation efficiency.

Its pseudo-label filtering mechanism is **Entropy-based Dynamic Filtering (EDF)**. EDF operates at two levels. At the image level, it computes local and global uncertainty. The local foreground probability is
$p_f = UF(Norm(P^I), \omega \times \omega)$ with $\omega = 7$, the local entropy is
$$E_{local} = - p_f \log p_f - p_b \log p_b,$$
and the image-level uncertainty is the fraction of pixels satisfying $E_{local} > 0.5 E_{global}$. Retention uses the threshold $\tau_\alpha = 0.3$: an unlabeled image is retained if $u_\alpha < \tau_\alpha$. At the pixel level, EDF attenuates uncertain pseudo-labels through
$$s(p) = 0.5 + 0.5(1 - E_{local}(p))^k,$$
with $k=1$, and forms an entropy-weighted pseudo-label
$$P^E = P^I \cdot s.$$
Retained samples are then ranked by mean local entropy and expanded according to a curriculum in which the top-$x$ lowest-entropy samples are selected first, with $x$ increasing over training stages.

SAM enters through **Domain Prompt-guided Mutual Correction (DPC)**. The pseudo-mask $P^E$ is binarized into connected contour regions, extremely small regions are filtered, and the remaining components yield a hybrid prompt set. Each component contributes a minimum axis-aligned bounding rectangle
$$Prompt_B(i) = [x_{min}, y_{min}, x_{max}, y_{max}],$$
and a point prompt derived from its geometric center if the center lies inside the mask, or from **AxialS** if the center falls outside an irregular or incomplete region. The resulting hybrid set is
$$Prompt_{P-B} = Prompt_P \cup Prompt_B.$$
SAM is run in inference mode with **ViT-H** weights and frozen parameters:
$$P^S = SAM(Image, Prompt_{P-B}).$$
Mutual correction then fuses the EDF-weighted pseudo-label with the SAM output:
$$P^C = fuse(P^E, P^S) = 0.5 \cdot P^E + 0.5 \cdot P^S.$$
The corrected pseudo-label $P^C$ becomes the final pseudo-label $\hat Y$ used to augment training.

The training objective is a hybrid loss
$$L_{total} = L_s + \alpha L_{dice} + \beta L_{UAL},$$
where $L_s = L_{wbce} + L_{wiou}$, $\alpha = 4$, $\beta = 2$, and $\lambda_{ual}$ decreases with the learning rate. The Dice term is
$$L_{dice} = 1 - \frac{2 \cdot \sum (Pred^p \cdot Target^p) + S}{\sum Pred^p + \sum Target^p + S}, \quad p=2,$$
and the uncertainty-aware loss is
$$L_{UAL} = \lambda_{ual} \cdot \sum [Pred \log Pred + (1 - Pred)\log(1 - Pred)].$$
The paper states that the same loss is applied to both labeled data and pseudo-labeled unlabeled data.

Architecturally, the reported implementation uses a **ConvNeXt-B backbone**, a **Frequency Transformer (FT)**, **ASPP** on the top-level feature, and a **reverse fusion decoder with reverse mask blocks**. Training uses **random flipping, rotation, boundary cropping**, input resolution **384 × 384**, **Adam**, supervised warm-up from **$1e^{-7}$ to $1e^{-4}$ over 10 epochs** followed by cosine annealing to **$1e^{-7}$ over 150 epochs**, then semi-supervised training with the learning rate reset to **$1e^{-4}$**, EDF+DPC expansion every **20 epochs**, and cosine annealing to **$1e^{-7}$ over 300 epochs**. The batch size is **8**.

## 3. SAMST / ST-SAM for remote sensing semantic segmentation

In remote sensing semantic segmentation, ST-SAM is presented as **SAMST**, a semi-supervised method for exploiting large pools of unlabeled aerial imagery under resolution variability and inconsistent land-cover category definitions. The method alternates between training a supervised segmentation model and refining its pseudo-labels through a **SAM-based Pseudo-label Refiner**, with one full iteration used in the reported experiments [2507.11994].

The initial model $M$ is trained on the labeled set $D_L$ only. For each unlabeled patch in $D_U$, the model outputs per-pixel logits $z_i(u)$ and class probabilities $p(i \mid u)$ via softmax. These probabilities drive three refiner modules: the **Threshold Filter Module (TFM)**, the **Prompt Generation Module (PGM)**, and the **Label Refinement Module (LRM)**.

TFM performs confidence gating. Let
$$i^*(u) = \arg\max_i p(i \mid u), \qquad \mathcal{P}_\tau = \{u : p(i^*(u)\mid u) \ge \tau_{i^*(u)}\}.$$
Then the pseudo-label is
$$\hat y(u) = i^*(u) \ \text{if}\ u \in \mathcal{P}_\tau,\ \text{else}\ 255.$$
The ignore label is therefore an explicit part of the filtering process. The paper notes that entropy-based filtering is optional, with
$$H(u) = -\sum_i p(i \mid u)\log p(i \mid u),$$
but the reported implementation uses **max probability thresholding per class** rather than entropy.

PGM converts accepted pseudo-label regions into SAM prompts. For each class $i$, connected components $A_i^k$ are extracted using **8-connectivity**. For each component, an axis-aligned box $B_i^k$ is computed and expanded by **$B_n$ pixels**. Positive points are sampled inside $A_i^k$ subject to
$$p(i \mid u_r) \ge T_p, \qquad N_{P_m}(u_r) \subset A_i^k,$$
while negative points are sampled from $B_i^k \setminus A_i^k$ subject to
$$\exists j \ne i \text{ with } p(j \mid v_r) \ge T_n, \qquad N_{N_m}(v_r) \subset (B_i^k \setminus A_i^k).$$
These prompts are then fed to SAM as one box plus positive and negative points for each connected region.

LRM receives the SAM masks $m_i^k(u) \in \{0,1\}$, applies hole filling to obtain $\bar m_i^k(u)$, and performs **probability-guided stitching**. With
$$j^*(u) = \arg\max_j p(j \mid u), \qquad p_{j^*}(u) = p(j^*(u)\mid u),$$
the update rule assigns class $i$ to pixel $u$ only if a refined SAM mask covers that pixel and either
$$j^*(u)=i \ \text{and}\ p(i \mid u)\ge t_c,$$
or
$$j^*(u)\ne i \ \text{and}\ p_{j^*}(u)\le t_o.$$
If multiple classes satisfy the condition, the chosen label is the one maximizing $p(i \mid u)$; otherwise the pixel remains **255**. This LRM step is the semantic reconciliation mechanism that aligns SAM’s geometric masks with the segmentation model’s class probabilities.

The retraining objective is a composite weighted cross-entropy. The labeled loss is
$$\mathcal{L}_{sup} = \frac{1}{|D_L|}\sum_{(x,y)\in D_L}\frac{1}{|\Omega_x|}\sum_{u \in \Omega_x}\left[-\sum_{i=1}^C w_{\ell,i}\mathbf{1}[y(u)=i]\log p(i\mid u)\right],$$
the pseudo-labeled loss is
$$\mathcal{L}_{unsup} = \frac{1}{|D_U'|}\sum_{(x,\hat y^{ref})\in D_U'}\frac{1}{|\Omega_x|}\sum_{u\in\Omega_x,\hat y^{ref}(u)\ne255}\left[-\sum_{i=1}^C w_{p,i}\mathbf{1}[\hat y^{ref}(u)=i]\log p(i\mid u)\right],$$
and the total objective is
$$\mathcal{L} = \mathcal{L}_{sup} + \alpha \mathcal{L}_{unsup}.$$

The backbone is a **Swin Transformer**, described as a hierarchical transformer with shifted window multi-head self-attention. Multi-scale features feed a **UperHead** decoder, and an auxiliary **FCNHead** provides intermediate supervision. Large Potsdam images of **6000 × 6000** at **5 cm/pixel** are tiled into **512 × 512 non-overlapping patches**. The implementation uses **PyTorch**, **eight NVIDIA A40 GPUs**, **batch size 4 per GPU**, **AdamW**, **initial LR $6e^{-5}$**, **weight decay 0.01**, and **linear decay with 1,500-iteration warmup**. Each stage runs for **80,000 iterations**, the best checkpoint is selected by validation for pseudo-label generation, and **Teacher-student/EMA is not used**.

## 4. Shared algorithmic structure and task-specific divergence

Despite the domain difference, the two frameworks instantiate a closely related sequence: initialize a supervised model on scarce labels, generate pseudo-labels on unlabeled data, filter or reweight them, transform them into SAM prompts, fuse or stitch SAM masks with model confidence, and retrain on the augmented set. In both cases, SAM is used in **inference-only** mode rather than being fine-tuned [2507.23307; 2507.11994].

| Component | ST-SAM for SSCOD | ST-SAM/SAMST for remote sensing |
|---|---|---|
| Trainable learner | Single-model COD network | Swin Transformer + UperHead + FCNHead |
| Pseudo-label curation | EDF with image-level retention and pixel-level entropy weighting | TFM with class-specific thresholds and ignore label 255 |
| SAM prompts | Hybrid points + boxes from pseudo-mask regions | Boxes + positive/negative points from connected components |
| SAM correction | $P^C = 0.5(P^E + P^S)$ | Probability-guided stitching with $t_c$ and $t_o$ |
| Iterative schedule | Expansion every 20 epochs, low→high entropy curriculum | One reported iteration, further rounds optional |

The main divergence lies in how each task formulates semantic consistency. In camouflaged object detection, the problem is binary or foreground-centric, so the correction step is an equal-weight fusion of the model-derived pseudo-label and the SAM output. In remote sensing, the problem is multi-class with taxonomy mismatch and “stuff” categories, so the correction step must resolve inter-class overlaps and explicitly preserve an ignore state. This suggests that the shared ST-SAM principle is stable across tasks, while the refinement operator is domain-specific.

## 5. Empirical results and comparative position

For semi-supervised camouflaged object detection, the training pool is **CAMO-train (1,000)** plus **COD10K-train (3,040)**, totaling **4,040 training images**. Test evaluation uses **CHAMELEON (76)**, **NC4K (4,121)**, **CAMO-test (250)**, and **COD10K-test (2,026)**. Labeled fractions are **1\%**, **5\%**, **10\%**, and **20\%**, with the remainder unlabeled. In the **1\%** setting, the paper states **$M \approx 40$ labeled images** and **$N \approx 4,000$ unlabeled images** [2507.23307].

At **1\% labeled**, the reported “Ours −1%” results are: **CAMO** with $E_\xi = 0.879$, $S_\alpha = 0.819$, $F_\beta^\omega = 0.753$, $F_\beta = 0.804$, $MAE = 0.070$; **CHAMELEON** with $E_\xi = 0.920$, $S_\alpha = 0.848$, $F_\beta^\omega = 0.778$, $F_\beta = 0.815$, $MAE = 0.038$; **COD10K** with $E_\xi = 0.884$, $S_\alpha = 0.824$, $F_\beta^\omega = 0.713$, $F_\beta = 0.737$, $MAE = 0.032$; and **NC4K** with $E_\xi = 0.907$, $S_\alpha = 0.855$, $F_\beta^\omega = 0.795$, $F_\beta = 0.830$, $MAE = 0.043$. Against **CamoTeacher (ECCV’24)** on COD10K with **1\% labels**, ST-SAM reports $F_\beta^\omega$ **0.713 vs 0.517** and $MAE$ **0.032 vs 0.062**. Against **SSCOD (ACM MM’24)** on COD10K with **1\% labels**, it reports $F_\beta^\omega$ **0.713 vs 0.537** and $MAE$ **0.032 vs 0.057**; the paper also states that ST-SAM with **1\% labels** can outperform SSCOD with **10\% labels** on multiple datasets, including **COD10K**, where $F_\beta^\omega$ is **0.713 vs 0.639**. With **20\% labels**, the reported values are: **CAMO** $E_\xi 0.890$, $S_\alpha 0.844$, $F_\beta^\omega 0.779$, $F_\beta 0.809$, $MAE 0.058$; **CHAMELEON** $E_\xi 0.926$, $S_\alpha 0.876$, $F_\beta^\omega 0.804$, $F_\beta 0.818$, $MAE 0.032$; **COD10K** $E_\xi 0.874$, $S_\alpha 0.837$, $F_\beta^\omega 0.713$, $F_\beta 0.717$, $MAE 0.030$; and **NC4K** $E_\xi 0.911$, $S_\alpha 0.874$, $F_\beta^\omega 0.807$, $F_\beta 0.824$, $MAE 0.037$.

The ablation study for camouflaged object detection is unusually detailed. On **COD10K / NC4K** in $F_\beta^\omega$, the **baseline (only $D_l$)** gives **0.517 / 0.611**, **+ vanilla self-training** gives **0.535 / 0.628**, **+ EDF** gives **0.676 / 0.748**, **+ DPC** gives **0.655 / 0.740**, and **+ EDF + DPC** gives **0.713 / 0.795**, which is the best setting. For EDF scheduling, **Expand all once** yields **0.621 / 0.737 / 0.676** on **COD10K / NC4K / CHAMELEON**, **Equal-ratio expansion** yields **0.682 / 0.778 / 0.753**, and the chosen **Epoch-dynamic** schedule yields **0.713 / 0.795 / 0.753**. For learning order, **High→Low entropy first collapses (0.525 on COD10K)**, **Random** reaches **0.710**, and **Low→High** reaches **0.713**. For DPC prompt design on **COD10K / NC4K**, **Points only** gives **0.645 / 0.742**, **Box only** gives **0.701 / 0.795**, and the chosen **Hybrid P–B** gives **0.713 / 0.795**. Fusion by **intersection** gives **0.581**, by **union** gives **0.622**, and by **ratio/average 0.5–0.5** gives **0.713** on COD10K. Scalability tests show that replacing the backbone with **PRNet** yields **0.653 / 0.741** on COD10K / NC4K, and replacing the loss with **BCE+IoU** yields **0.698 / 0.798**.

For remote sensing, the evaluation uses the **Potsdam dataset**, consisting of **38 aerial images (6000 × 6000, 5 cm/pixel)** with **6 classes: Impervious Surface, Building, Low Vegetation, Tree, Car, Clutter**. The split is **24 images for training** and **14 for validation/testing**, tiled into **3,456 training patches** and **2,016 testing patches**. In the semi-supervised setting, **1/32 of training patches labeled (108)** and **3,348 unlabeled** are used, and evaluation reports **mIoU**, **OA**, **mF1**, and **per-class F1** [2507.11994].

On Potsdam test under the **1/32 labeled** regime, the reported results are:

| Method | Reported aggregate metrics | Reported per-class F1 |
|---|---|---|
| Baseline | mIoU 67.00, OA 93.48, mF1 78.56 | Clutter 46.43, Car 87.76, Tree 76.03, Low Veg. 80.74, Building 92.61, Surface 87.83 |
| LSST | mIoU 60.82, OA 80.57, mF1 74.15 | Clutter 42.04, Car 76.87, Tree 78.39, Low Veg. 77.21, Building 87.34, Surface 83.04 |
| ST++ | mIoU 64.89, OA 83.83, mF1 75.37 | Clutter 23.32, Car 88.18, Tree 82.93, Low Veg. 81.64, Building 89.39, Surface 86.78 |
| ClassHyPer | mIoU 66.58, OA 83.62, mF1 78.23 | Clutter 43.55, Car 85.71, Tree 82.69, Low Veg. 80.42, Building 90.62, Surface 86.40 |
| SAMST | mIoU 70.80, OA 86.44, mF1 81.65 | Clutter 54.64, Car 89.56, Tree 83.56, Low Veg. 79.05, Building 93.80, Surface 89.28 |

The reported gains for SAMST are **+3.80 mIoU over Baseline**, **+9.98 over LSST**, **+5.91 over ST++**, and **+4.22 over ClassHyPer**. The paper also notes that **OA is higher than other SSL baselines, though lower than the Baseline model in this table**. Per-class behavior is non-uniform: **Clutter** improves by **+8.21 over Baseline**, **Building** and **Surface** also improve, while **Low Vegetation slightly decreases**. The paper does **not report ablation studies**, but qualitatively attributes the gains to the complementary action of TFM, PGM, and LRM.

Taken together, the reported evidence places ST-SAM in a specific comparative niche. It is not merely a pseudo-labeling strategy, nor merely a prompt engineering method for SAM. Its empirical edge is associated with the combination of selective pseudo-label admission, domain-aware prompting, and a correction rule that restricts SAM to a pseudo-label refinement role rather than treating SAM output as a standalone replacement.

## 6. Limitations, failure modes, and extensions

Both variants explicitly describe failure modes. In camouflaged object detection, **extremely low-contrast micro-objects** and **heavy occlusions** can still produce incomplete $P^E$, which in turn limits prompt quality. In scattered multi-instance scenes, boxes can become overly large. The method is also sensitive to **$\tau_\alpha$**, the binarization used to extract $Mask_i$, and small-region filtering. The paper states that the method is robust in reported settings but may require light tuning under domain shifts. It further acknowledges a residual dependence on pseudo-label quality: if contours are **grossly mislocalized**, the prompts may misguide SAM, even though center-safe points and hybrid prompts reduce that risk [2507.23307].

In remote sensing, the limitations are framed somewhat differently. The method is sensitive to **prompt counts and prompt placement** in cluttered scenes, and **aggressive $\tau/t_c/t_o$** may remove valid pixels while lenient settings admit noise. **Class-specific tuning is necessary**. The paper also notes a semantic asymmetry in SAM’s behavior: SAM segments **“things”** well, but some **“stuff”** classes, particularly **low vegetation**, may be less precise. Hyperparameters such as **$B_n$, $P_p$, $P_n$, $T_p$, $P_m$, $T_n$, $N_m$, $t_c$, $t_o$** are described as dataset- and class-dependent and set empirically. The absence of ablation studies leaves the relative sensitivity of these components less directly quantified than in the camouflaged object detection paper [2507.11994].

The suggested extensions also follow the same shared logic. For remote sensing, the paper proposes **uncertainty-aware filtering (entropy/MC-dropout)**, **multi-scale prompting and tiling strategies**, **consistency regularization (e.g., Mean Teacher/EMA)**, and **CRF or transformer-based boundary refinement post-SAM**. For camouflaged object detection, the paper identifies **multi-modal prompts**, **temporal prompts for video**, and **uncertainty-calibrated SAM ensemble prompts** as plausible extensions. This suggests that ST-SAM is best understood not as a closed algorithm, but as a design pattern for annotation-efficient segmentation in which SAM is inserted into self-training as a frozen geometric prior while the task model remains the semantic anchor.

Source: https://www.emergentmind.com/topics/st-sam