ST-SAM: Self-Training with SAM Refinement

• ST-SAM is a self-training method that uses the Segment Anything Model to refine noisy pseudo-labels in dense pixel-level tasks.
• It employs entropy-based dynamic filtering and domain-specific prompting to improve boundary quality and mitigate prediction bias.
• The framework adapts to tasks like camouflaged object detection and remote sensing segmentation, achieving superior metrics with minimal labeled data.

ST-SAM denotes a self-training formulation that integrates the Segment Anything Model (SAM) into semi-supervised dense prediction by using model-generated pseudo-labels as the basis for SAM-guided refinement. In the 2025 literature, the name appears explicitly in “ST-SAM: SAM-Driven Self-Training Framework for Semi-Supervised Camouflaged Object Detection” and equivalently in remote sensing semantic segmentation as “ST-SAM/SAMST (Self-Training with SAM),” where a supervised segmentation model alternates with a SAM-based Pseudo-label Refiner (Hu et al., 31 Jul 2025, Yin et al., 16 Jul 2025). In both usages, the central premise is the same: dense pixel annotation is scarce, unlabeled imagery is abundant, and naive self-training is limited by noisy pseudo-labels and weak boundary quality. ST-SAM addresses this by coupling confidence-based pseudo-label curation with SAM’s zero-shot mask generation.

1. Terminology, problem setting, and naming overlap

The term ST-SAM is not confined to a single task or a single fixed architecture. In semi-supervised camouflaged object detection, ST-SAM is introduced as a SAM-Driven Self-Training Framework for Semi-Supervised Camouflaged Object Detection. In remote sensing semantic segmentation, the same idea is described as ST-SAM/SAMST (Self-Training with SAM), with “SAMST” emphasizing SAM’s role in self-training and “ST-SAM” denoting the same integration pattern (Hu et al., 31 Jul 2025, Yin et al., 16 Jul 2025).

The two problem domains differ materially. In semi-supervised camouflaged object detection, the difficulty arises from large scale variations, complex contours, and textures that are often indistinguishable from the background, making high-quality pixel-level annotations expensive and scarce. In remote sensing semantic segmentation, the motivating factors are label scarcity, resolution variability, and inconsistent land-cover definitions across sources, all of which hinder direct supervised scaling. Despite these differences, both formulations are organized around the same failure mode of standard pseudo-labeling: early pseudo-labels are informative enough to expand supervision, but sufficiently noisy to contaminate later optimization if they are used without filtration and geometric correction.

A common misconception is that ST-SAM names a single architectural template. The published descriptions indicate the opposite. The camouflaged object detection variant is explicitly model-agnostic and is demonstrated with a standard encoder–decoder using a ConvNeXt-B backbone, while the remote sensing variant uses a Swin Transformer with UperHead and an auxiliary FCNHead. What is shared is not a backbone, but a training logic in which pseudo-labels are first filtered, then converted into SAM prompts, and finally reinjected into model training after SAM-assisted correction.

2. ST-SAM for semi-supervised camouflaged object detection

In semi-supervised camouflaged object detection, the labeled and unlabeled sets are denoted $D_l = \{(x_i^l, y_i)\}_{i=1}^M$ and $D_u = \{x_j^u\}_{j=1}^N$ with $M \ll N$, and the single segmentation model is written as $f_\theta : x \rightarrow \mathrm{Pred}$, where $\mathrm{Pred}$ is a soft foreground probability map. The initial pseudo-label for an unlabeled image is $P^I = f_\theta(x)$ (Hu et al., 31 Jul 2025).

The framework is motivated by explicit limitations of teacher–student SSCOD. Under very scarce supervision, multi-network pipelines are described as suffering from prediction bias, error propagation, and increased computational overhead and scalability constraints. ST-SAM replaces that regime with single-model self-training, with the stated goal of fundamentally circumventing inter-model prediction bias while maintaining annotation efficiency.

Its pseudo-label filtering mechanism is Entropy-based Dynamic Filtering (EDF). EDF operates at two levels. At the image level, it computes local and global uncertainty. The local foreground probability is $p_f = UF(Norm(P^I), \omega \times \omega)$ with $\omega = 7$, the local entropy is

$E_{local} = - p_f \log p_f - p_b \log p_b,$

and the image-level uncertainty is the fraction of pixels satisfying $E_{local} > 0.5 E_{global}$. Retention uses the threshold $D_u = \{x_j^u\}_{j=1}^N$0: an unlabeled image is retained if $D_u = \{x_j^u\}_{j=1}^N$1. At the pixel level, EDF attenuates uncertain pseudo-labels through

$D_u = \{x_j^u\}_{j=1}^N$2

with $D_u = \{x_j^u\}_{j=1}^N$3, and forms an entropy-weighted pseudo-label

$D_u = \{x_j^u\}_{j=1}^N$4

Retained samples are then ranked by mean local entropy and expanded according to a curriculum in which the top-$D_u = \{x_j^u\}_{j=1}^N$5 lowest-entropy samples are selected first, with $D_u = \{x_j^u\}_{j=1}^N$6 increasing over training stages.

SAM enters through Domain Prompt-guided Mutual Correction (DPC). The pseudo-mask $D_u = \{x_j^u\}_{j=1}^N$7 is binarized into connected contour regions, extremely small regions are filtered, and the remaining components yield a hybrid prompt set. Each component contributes a minimum axis-aligned bounding rectangle

$D_u = \{x_j^u\}_{j=1}^N$8

and a point prompt derived from its geometric center if the center lies inside the mask, or from AxialS if the center falls outside an irregular or incomplete region. The resulting hybrid set is

$D_u = \{x_j^u\}_{j=1}^N$9

SAM is run in inference mode with ViT-H weights and frozen parameters:

$M \ll N$0

Mutual correction then fuses the EDF-weighted pseudo-label with the SAM output:

$M \ll N$1

The corrected pseudo-label $M \ll N$2 becomes the final pseudo-label $M \ll N$3 used to augment training.

The training objective is a hybrid loss

$M \ll N$4

where $M \ll N$5, $M \ll N$6, $M \ll N$7, and $M \ll N$8 decreases with the learning rate. The Dice term is

$M \ll N$9

and the uncertainty-aware loss is

$f_\theta : x \rightarrow \mathrm{Pred}$0

The paper states that the same loss is applied to both labeled data and pseudo-labeled unlabeled data.

Architecturally, the reported implementation uses a ConvNeXt-B backbone, a Frequency Transformer (FT), ASPP on the top-level feature, and a reverse fusion decoder with reverse mask blocks. Training uses random flipping, rotation, boundary cropping, input resolution 384 × 384, Adam, supervised warm-up from $f_\theta : x \rightarrow \mathrm{Pred}$1 to $f_\theta : x \rightarrow \mathrm{Pred}$2 over 10 epochs followed by cosine annealing to $f_\theta : x \rightarrow \mathrm{Pred}$3 over 150 epochs, then semi-supervised training with the learning rate reset to $f_\theta : x \rightarrow \mathrm{Pred}$4, EDF+DPC expansion every 20 epochs, and cosine annealing to $f_\theta : x \rightarrow \mathrm{Pred}$5 over 300 epochs. The batch size is 8.

3. SAMST / ST-SAM for remote sensing semantic segmentation

In remote sensing semantic segmentation, ST-SAM is presented as SAMST, a semi-supervised method for exploiting large pools of unlabeled aerial imagery under resolution variability and inconsistent land-cover category definitions. The method alternates between training a supervised segmentation model and refining its pseudo-labels through a SAM-based Pseudo-label Refiner, with one full iteration used in the reported experiments (Yin et al., 16 Jul 2025).

The initial model $f_\theta : x \rightarrow \mathrm{Pred}$6 is trained on the labeled set $f_\theta : x \rightarrow \mathrm{Pred}$7 only. For each unlabeled patch in $f_\theta : x \rightarrow \mathrm{Pred}$8, the model outputs per-pixel logits $f_\theta : x \rightarrow \mathrm{Pred}$9 and class probabilities $\mathrm{Pred}$0 via softmax. These probabilities drive three refiner modules: the Threshold Filter Module (TFM), the Prompt Generation Module (PGM), and the Label Refinement Module (LRM).

TFM performs confidence gating. Let

$\mathrm{Pred}$1

Then the pseudo-label is

$\mathrm{Pred}$2

The ignore label is therefore an explicit part of the filtering process. The paper notes that entropy-based filtering is optional, with

$\mathrm{Pred}$3

but the reported implementation uses max probability thresholding per class rather than entropy.

PGM converts accepted pseudo-label regions into SAM prompts. For each class $\mathrm{Pred}$4, connected components $\mathrm{Pred}$5 are extracted using 8-connectivity. For each component, an axis-aligned box $\mathrm{Pred}$6 is computed and expanded by $\mathrm{Pred}$7 pixels. Positive points are sampled inside $\mathrm{Pred}$8 subject to

$\mathrm{Pred}$9

while negative points are sampled from $P^I = f_\theta(x)$0 subject to

$P^I = f_\theta(x)$1

These prompts are then fed to SAM as one box plus positive and negative points for each connected region.

LRM receives the SAM masks $P^I = f_\theta(x)$2, applies hole filling to obtain $P^I = f_\theta(x)$3, and performs probability-guided stitching. With

$P^I = f_\theta(x)$4

the update rule assigns class $P^I = f_\theta(x)$5 to pixel $P^I = f_\theta(x)$6 only if a refined SAM mask covers that pixel and either

$P^I = f_\theta(x)$7

or

$P^I = f_\theta(x)$8

If multiple classes satisfy the condition, the chosen label is the one maximizing $P^I = f_\theta(x)$9; otherwise the pixel remains 255. This LRM step is the semantic reconciliation mechanism that aligns SAM’s geometric masks with the segmentation model’s class probabilities.

The retraining objective is a composite weighted cross-entropy. The labeled loss is

$p_f = UF(Norm(P^I), \omega \times \omega)$0

the pseudo-labeled loss is

$p_f = UF(Norm(P^I), \omega \times \omega)$1

and the total objective is

$p_f = UF(Norm(P^I), \omega \times \omega)$2

The backbone is a Swin Transformer, described as a hierarchical transformer with shifted window multi-head self-attention. Multi-scale features feed a UperHead decoder, and an auxiliary FCNHead provides intermediate supervision. Large Potsdam images of 6000 × 6000 at 5 cm/pixel are tiled into 512 × 512 non-overlapping patches. The implementation uses PyTorch, eight NVIDIA A40 GPUs, batch size 4 per GPU, AdamW, initial LR $p_f = UF(Norm(P^I), \omega \times \omega)$3, weight decay 0.01, and linear decay with 1,500-iteration warmup. Each stage runs for 80,000 iterations, the best checkpoint is selected by validation for pseudo-label generation, and Teacher-student/EMA is not used.

4. Shared algorithmic structure and task-specific divergence

Despite the domain difference, the two frameworks instantiate a closely related sequence: initialize a supervised model on scarce labels, generate pseudo-labels on unlabeled data, filter or reweight them, transform them into SAM prompts, fuse or stitch SAM masks with model confidence, and retrain on the augmented set. In both cases, SAM is used in inference-only mode rather than being fine-tuned (Hu et al., 31 Jul 2025, Yin et al., 16 Jul 2025).

Component	ST-SAM for SSCOD	ST-SAM/SAMST for remote sensing
Trainable learner	Single-model COD network	Swin Transformer + UperHead + FCNHead
Pseudo-label curation	EDF with image-level retention and pixel-level entropy weighting	TFM with class-specific thresholds and ignore label 255
SAM prompts	Hybrid points + boxes from pseudo-mask regions	Boxes + positive/negative points from connected components
SAM correction	$p_f = UF(Norm(P^I), \omega \times \omega)$4	Probability-guided stitching with $p_f = UF(Norm(P^I), \omega \times \omega)$5 and $p_f = UF(Norm(P^I), \omega \times \omega)$6
Iterative schedule	Expansion every 20 epochs, low→high entropy curriculum	One reported iteration, further rounds optional

The main divergence lies in how each task formulates semantic consistency. In camouflaged object detection, the problem is binary or foreground-centric, so the correction step is an equal-weight fusion of the model-derived pseudo-label and the SAM output. In remote sensing, the problem is multi-class with taxonomy mismatch and “stuff” categories, so the correction step must resolve inter-class overlaps and explicitly preserve an ignore state. This suggests that the shared ST-SAM principle is stable across tasks, while the refinement operator is domain-specific.

5. Empirical results and comparative position

For semi-supervised camouflaged object detection, the training pool is CAMO-train (1,000) plus COD10K-train (3,040), totaling 4,040 training images. Test evaluation uses CHAMELEON (76), NC4K (4,121), CAMO-test (250), and COD10K-test (2,026). Labeled fractions are 1\%, 5\%, 10\%, and 20\%, with the remainder unlabeled. In the 1\% setting, the paper states $p_f = UF(Norm(P^I), \omega \times \omega)$7 labeled images and $p_f = UF(Norm(P^I), \omega \times \omega)$8 unlabeled images (Hu et al., 31 Jul 2025).

At 1\% labeled, the reported “Ours −1%” results are: CAMO with $p_f = UF(Norm(P^I), \omega \times \omega)$9, $\omega = 7$0, $\omega = 7$1, $\omega = 7$2, $\omega = 7$3; CHAMELEON with $\omega = 7$4, $\omega = 7$5, $\omega = 7$6, $\omega = 7$7, $\omega = 7$8; COD10K with $\omega = 7$9, $E_{local} = - p_f \log p_f - p_b \log p_b,$0, $E_{local} = - p_f \log p_f - p_b \log p_b,$1, $E_{local} = - p_f \log p_f - p_b \log p_b,$2, $E_{local} = - p_f \log p_f - p_b \log p_b,$3; and NC4K with $E_{local} = - p_f \log p_f - p_b \log p_b,$4, $E_{local} = - p_f \log p_f - p_b \log p_b,$5, $E_{local} = - p_f \log p_f - p_b \log p_b,$6, $E_{local} = - p_f \log p_f - p_b \log p_b,$7, $E_{local} = - p_f \log p_f - p_b \log p_b,$8. Against CamoTeacher (ECCV’24) on COD10K with 1\% labels, ST-SAM reports $E_{local} = - p_f \log p_f - p_b \log p_b,$9 0.713 vs 0.517 and $E_{local} > 0.5 E_{global}$0 0.032 vs 0.062. Against SSCOD (ACM MM’24) on COD10K with 1\% labels, it reports $E_{local} > 0.5 E_{global}$1 0.713 vs 0.537 and $E_{local} > 0.5 E_{global}$2 0.032 vs 0.057; the paper also states that ST-SAM with 1\% labels can outperform SSCOD with 10\% labels on multiple datasets, including COD10K, where $E_{local} > 0.5 E_{global}$3 is 0.713 vs 0.639. With 20\% labels, the reported values are: CAMO $E_{local} > 0.5 E_{global}$4, $E_{local} > 0.5 E_{global}$5, $E_{local} > 0.5 E_{global}$6, $E_{local} > 0.5 E_{global}$7, $E_{local} > 0.5 E_{global}$8; CHAMELEON $E_{local} > 0.5 E_{global}$9, $D_u = \{x_j^u\}_{j=1}^N$00, $D_u = \{x_j^u\}_{j=1}^N$01, $D_u = \{x_j^u\}_{j=1}^N$02, $D_u = \{x_j^u\}_{j=1}^N$03; COD10K $D_u = \{x_j^u\}_{j=1}^N$04, $D_u = \{x_j^u\}_{j=1}^N$05, $D_u = \{x_j^u\}_{j=1}^N$06, $D_u = \{x_j^u\}_{j=1}^N$07, $D_u = \{x_j^u\}_{j=1}^N$08; and NC4K $D_u = \{x_j^u\}_{j=1}^N$09, $D_u = \{x_j^u\}_{j=1}^N$10, $D_u = \{x_j^u\}_{j=1}^N$11, $D_u = \{x_j^u\}_{j=1}^N$12, $D_u = \{x_j^u\}_{j=1}^N$13.

The ablation study for camouflaged object detection is unusually detailed. On COD10K / NC4K in $D_u = \{x_j^u\}_{j=1}^N$14, the baseline (only $D_u = \{x_j^u\}_{j=1}^N$15) gives 0.517 / 0.611, + vanilla self-training gives 0.535 / 0.628, + EDF gives 0.676 / 0.748, + DPC gives 0.655 / 0.740, and + EDF + DPC gives 0.713 / 0.795, which is the best setting. For EDF scheduling, Expand all once yields 0.621 / 0.737 / 0.676 on COD10K / NC4K / CHAMELEON, Equal-ratio expansion yields 0.682 / 0.778 / 0.753, and the chosen Epoch-dynamic schedule yields 0.713 / 0.795 / 0.753. For learning order, High→Low entropy first collapses (0.525 on COD10K), Random reaches 0.710, and Low→High reaches 0.713. For DPC prompt design on COD10K / NC4K, Points only gives 0.645 / 0.742, Box only gives 0.701 / 0.795, and the chosen Hybrid P–B gives 0.713 / 0.795. Fusion by intersection gives 0.581, by union gives 0.622, and by ratio/average 0.5–0.5 gives 0.713 on COD10K. Scalability tests show that replacing the backbone with PRNet yields 0.653 / 0.741 on COD10K / NC4K, and replacing the loss with BCE+IoU yields 0.698 / 0.798.

For remote sensing, the evaluation uses the Potsdam dataset, consisting of 38 aerial images (6000 × 6000, 5 cm/pixel) with 6 classes: Impervious Surface, Building, Low Vegetation, Tree, Car, Clutter. The split is 24 images for training and 14 for validation/testing, tiled into 3,456 training patches and 2,016 testing patches. In the semi-supervised setting, 1/32 of training patches labeled (108) and 3,348 unlabeled are used, and evaluation reports mIoU, OA, mF1, and per-class F1 (Yin et al., 16 Jul 2025).

On Potsdam test under the 1/32 labeled regime, the reported results are:

Method	Reported aggregate metrics	Reported per-class F1
Baseline	mIoU 67.00, OA 93.48, mF1 78.56	Clutter 46.43, Car 87.76, Tree 76.03, Low Veg. 80.74, Building 92.61, Surface 87.83
LSST	mIoU 60.82, OA 80.57, mF1 74.15	Clutter 42.04, Car 76.87, Tree 78.39, Low Veg. 77.21, Building 87.34, Surface 83.04
ST++	mIoU 64.89, OA 83.83, mF1 75.37	Clutter 23.32, Car 88.18, Tree 82.93, Low Veg. 81.64, Building 89.39, Surface 86.78
ClassHyPer	mIoU 66.58, OA 83.62, mF1 78.23	Clutter 43.55, Car 85.71, Tree 82.69, Low Veg. 80.42, Building 90.62, Surface 86.40
SAMST	mIoU 70.80, OA 86.44, mF1 81.65	Clutter 54.64, Car 89.56, Tree 83.56, Low Veg. 79.05, Building 93.80, Surface 89.28

The reported gains for SAMST are +3.80 mIoU over Baseline, +9.98 over LSST, +5.91 over ST++, and +4.22 over ClassHyPer. The paper also notes that OA is higher than other SSL baselines, though lower than the Baseline model in this table. Per-class behavior is non-uniform: Clutter improves by +8.21 over Baseline, Building and Surface also improve, while Low Vegetation slightly decreases. The paper does not report ablation studies, but qualitatively attributes the gains to the complementary action of TFM, PGM, and LRM.

Taken together, the reported evidence places ST-SAM in a specific comparative niche. It is not merely a pseudo-labeling strategy, nor merely a prompt engineering method for SAM. Its empirical edge is associated with the combination of selective pseudo-label admission, domain-aware prompting, and a correction rule that restricts SAM to a pseudo-label refinement role rather than treating SAM output as a standalone replacement.

6. Limitations, failure modes, and extensions

Both variants explicitly describe failure modes. In camouflaged object detection, extremely low-contrast micro-objects and heavy occlusions can still produce incomplete $D_u = \{x_j^u\}_{j=1}^N$16, which in turn limits prompt quality. In scattered multi-instance scenes, boxes can become overly large. The method is also sensitive to $D_u = \{x_j^u\}_{j=1}^N$17, the binarization used to extract $D_u = \{x_j^u\}_{j=1}^N$18, and small-region filtering. The paper states that the method is robust in reported settings but may require light tuning under domain shifts. It further acknowledges a residual dependence on pseudo-label quality: if contours are grossly mislocalized, the prompts may misguide SAM, even though center-safe points and hybrid prompts reduce that risk (Hu et al., 31 Jul 2025).

In remote sensing, the limitations are framed somewhat differently. The method is sensitive to prompt counts and prompt placement in cluttered scenes, and aggressive $D_u = \{x_j^u\}_{j=1}^N$19 may remove valid pixels while lenient settings admit noise. Class-specific tuning is necessary. The paper also notes a semantic asymmetry in SAM’s behavior: SAM segments “things” well, but some “stuff” classes, particularly low vegetation, may be less precise. Hyperparameters such as $D_u = \{x_j^u\}_{j=1}^N$20, $D_u = \{x_j^u\}_{j=1}^N$21, $D_u = \{x_j^u\}_{j=1}^N$22, $D_u = \{x_j^u\}_{j=1}^N$23, $D_u = \{x_j^u\}_{j=1}^N$24, $D_u = \{x_j^u\}_{j=1}^N$25, $D_u = \{x_j^u\}_{j=1}^N$26, $D_u = \{x_j^u\}_{j=1}^N$27, $D_u = \{x_j^u\}_{j=1}^N$28 are described as dataset- and class-dependent and set empirically. The absence of ablation studies leaves the relative sensitivity of these components less directly quantified than in the camouflaged object detection paper (Yin et al., 16 Jul 2025).

The suggested extensions also follow the same shared logic. For remote sensing, the paper proposes uncertainty-aware filtering (entropy/MC-dropout), multi-scale prompting and tiling strategies, consistency regularization (e.g., Mean Teacher/EMA), and CRF or transformer-based boundary refinement post-SAM. For camouflaged object detection, the paper identifies multi-modal prompts, temporal prompts for video, and uncertainty-calibrated SAM ensemble prompts as plausible extensions. This suggests that ST-SAM is best understood not as a closed algorithm, but as a design pattern for annotation-efficient segmentation in which SAM is inserted into self-training as a frozen geometric prior while the task model remains the semantic anchor.