Selective Semi-supervised Fine-tuning

• Selective semi-supervised fine-tuning is a method that updates pretrained models by integrating a small labeled set with a larger unlabeled pool using selective sample, pseudo-label, or parameter choices.
• It employs criteria such as feature-space distance, entropy thresholds, and judge-based gating to curate high-quality pseudo-labels and reduce noise during adaptation.
• This approach is applied in domains like semi-supervised domain adaptation, LLM alignment, medical segmentation, and NER, demonstrating significant performance improvements and theoretical benefits.

Selective semi-supervised fine-tuning denotes a family of adaptation procedures in which a pretrained or warm-started model is updated with a small labeled set and a larger unlabeled pool, while the optimization is made selective with respect to samples, pseudo-labels, parameters, latent subspaces, or even token positions. In the cited literature, this label covers semi-supervised domain adaptation, foundation-model adaptation, LLM alignment, ASR, medical segmentation, NER with partial annotation, and theoretically grounded target-domain fine-tuning from multiple adaptive starts (Kim et al., 2021, Luo et al., 2024, Ha et al., 19 Jul 2025).

1. Problem formulations and scope

A canonical formulation appears in semi-supervised domain adaptation, where the source domain is $D_s=\{(x_s,y_s)\}$, the target domain has a few labeled samples $D_t^L=\{(x_t^l,y_t^l)\}$, typically 1-shot or 3-shot per class, and a larger unlabeled set $D_t^U=\{x_t^u\}$. The model is decomposed into a feature extractor $F(\cdot;\theta_F)$ and classifier $C(\cdot;\theta_C)$, with feature $\phi_\theta(x)=F(x;\theta_F)$ and prediction $p_\theta(x)=C(\phi_\theta(x);\theta_C)$. The target-oriented model is then fine-tuned by leveraging the small labeled target set together with selectively chosen pseudo labels for $D_t^U$ (Kim et al., 2021).

In LLM alignment, the same structure is written as a labeled set $D_{\text{labeled}}=\{(t_i,y_i)\}$ and an unlabeled set $D_{\text{unlabeled}}=\{t_j\}$. The objective is to propagate target-task knowledge from labeled data to unlabeled data, generate and select high-quality pseudo-responses, and fine-tune the LLM to yield an evolved model adapted to the target domain (Luo et al., 2024). In long-tailed semi-supervised learning with foundation models, the labeled set $D_t^L=\{(x_t^l,y_t^l)\}$0 is imbalanced, the unlabeled set $D_t^L=\{(x_t^l,y_t^l)\}$1 may not match the labeled distribution, and, in open-world settings, $D_t^L=\{(x_t^l,y_t^l)\}$2 may also contain out-of-distribution samples (Chen et al., 12 Sep 2025).

The same logic extends beyond ordinary labeled/unlabeled splits. In partially annotated NER corpora, token positions are partitioned into trusted indices $D_t^L=\{(x_t^l,y_t^l)\}$3 and unreliable or missing positions $D_t^L=\{(x_t^l,y_t^l)\}$4; many true entity tokens are marked as “O” despite being entity tokens. The fine-tuning problem then becomes selective both over tokens and over supervision type, because known spans are enforced while the remaining positions are trained through soft targets rather than hard negatives (Scherbakov et al., 2022). This suggests that selective semi-supervised fine-tuning is better understood as a family of constrained or filtered fine-tuning regimes than as a single algorithmic template.

2. Data selection and pseudo-label curation

The most direct form of selectivity is explicit sample selection for pseudo-labeling. In semi-supervised domain adaptation, unlabeled target samples are first assigned soft pseudo labels $D_t^L=\{(x_t^l,y_t^l)\}$5 and hard pseudo labels $D_t^L=\{(x_t^l,y_t^l)\}$6. Selection is then driven not by a confidence threshold but by feature-space distance to labeled target samples of the same predicted class:

$D_t^L=\{(x_t^l,y_t^l)\}$7

For each class $D_t^L=\{(x_t^l,y_t^l)\}$8, unlabeled targets with $D_t^L=\{(x_t^l,y_t^l)\}$9 are sorted by ascending $D_t^U=\{x_t^u\}$0, and the top $D_t^U=\{x_t^u\}$1 are selected, with default $D_t^U=\{x_t^u\}$2. Because $D_t^U=\{x_t^u\}$3 is fixed per class, the selected pseudo-labeled set is balanced across classes (Kim et al., 2021).

LLM methods often select by response quality rather than feature geometry. SemiEvol generates multiple candidate responses, aggregates them with a Self-Justify step, computes response entropy

$D_t^U=\{x_t^u\}$4

and keeps only pseudo-responses whose entropy is below a dynamic threshold $D_t^U=\{x_t^u\}$5 with $D_t^U=\{x_t^u\}$6 (Luo et al., 2024). Selective Self-Rehearsal uses an LLM-as-a-judge instead: if $D_t^U=\{x_t^u\}$7, the model-generated response is used as the training target; otherwise the gold response is retained. The loss becomes

$D_t^U=\{x_t^u\}$8

thereby turning pseudo-labeling into a judge-gated substitution problem rather than an unlabeled-data expansion problem (Gupta et al., 2024).

Speech and 3D medical imaging use other selection rules. In multi-domain ASR, pseudo-labeled utterances are filtered either by multi-model consensus, using average pairwise CER

$D_t^U=\{x_t^u\}$9

with retain-if $F(\cdot;\theta_F)$0, or by NER, retaining segments with at least one detected named entity (Carofilis et al., 5 Jun 2025). In Active Source-Free Domain Adaptation for volumetric medical segmentation, reliability is defined as

$F(\cdot;\theta_F)$1

where $F(\cdot;\theta_F)$2 is a foreground-aware confidence score and $F(\cdot;\theta_F)$3 is cosine distance to labeled anchors; the top $F(\cdot;\theta_F)$4 volumes are then selected, and low-confidence voxels within them are masked with $F(\cdot;\theta_F)$5 (Yang et al., 13 Sep 2025). In open-world long-tailed SSL, LoFT-OW first prefilters unlabeled data by zero-shot confidence with $F(\cdot;\theta_F)$6, then applies MSP-based filtering and a confidence split between hard pseudo-labels and soft consistency supervision (Chen et al., 12 Sep 2025).

3. Selective objectives, parameters, and representations

Selectivity is not limited to sample choice. In several methods it is embedded directly into the objective or into gradient routing. In content-style decomposition for vision foundation models, the backbone latent $F(\cdot;\theta_F)$7 is decomposed into content $F(\cdot;\theta_F)$8 and style $F(\cdot;\theta_F)$9, the supervised loss is

$C(\cdot;\theta_C)$0

and the semi-supervised part is

$C(\cdot;\theta_C)$1

Crucially, the backbone is updated only by supervised gradients, $C(\cdot;\theta_C)$2, while $C(\cdot;\theta_C)$3. This is selective semi-supervised fine-tuning at the parameter level: the backbone receives only supervised gradients; the content, style, decoder, and discriminator heads receive unlabeled regularization (Drozdova et al., 2024).

FineSSL makes selectivity class-aware and branch-aware. Balanced Margin Softmax computes a class-dependent margin from unlabeled learning pace,

$C(\cdot;\theta_C)$4

and uses

$C(\cdot;\theta_C)$5

Unlabeled samples are not hard-dropped in the main branch; instead they are weighted by an auxiliary classifier,

$C(\cdot;\theta_C)$6

while Decoupled Label Smoothing applies smoothing only in the detached auxiliary branch through

$C(\cdot;\theta_C)$7

This produces selective pressure across classes and across branches rather than only across samples (Gan et al., 2024).

Other methods make the objective selective through grouped contrast or through trainable-head restriction. Self-Tuning keeps a pretrained backbone, optimizes supervised cross-entropy on labeled data, and replaces unsupervised CE with Pseudo Group Contrast, where an unlabeled sample with pseudo-label $C(\cdot;\theta_C)$8 contrasts its query against $C(\cdot;\theta_C)$9 and against all other class queues. The resulting grouped positives make low-quality pseudo-labels contribute small gradients without any hard confidence threshold (Wang et al., 2021). TwinTURBO freezes the foundation backbone and trains only a shallow transformer, projector, predictor, and critic, combining a task-space mutual-information bound with a latent-space InfoNCE-style loss; the total objective is

$\phi_\theta(x)=F(x;\theta_F)$0

with $\phi_\theta(x)=F(x;\theta_F)$1 itself combining conditional and marginal cross-entropy and a JSD-based critic (Quétant et al., 10 Mar 2025). This suggests that selective semi-supervised fine-tuning often refers as much to which parameters are allowed to move and which losses are allowed to reach them as to which samples are admitted.

A related pre-fine-tuning form appears in task-specific masking for LLMs. A word score

$\phi_\theta(x)=F(x;\theta_F)$2

is computed from seed word lists via a linear SVM in Word2Vec space, mapped to a masking probability $\phi_\theta(x)=F(x;\theta_F)$3 through step, linear, or exponential functions, and used in a short MLM bridge stage before standard supervised fine-tuning (Lad et al., 2022). Here the selective unit is not the sample or the parameter, but the token position.

4. Algorithmic workflows

Many selective semi-supervised fine-tuning pipelines share a multi-stage schedule. In semi-supervised domain adaptation, the baseline warm-up uses Minimax Entropy to obtain $\phi_\theta(x)=F(x;\theta_F)$4; a selected set $\phi_\theta(x)=F(x;\theta_F)$5 is then built by feature-space distance; soft pseudo labels are initialized as $\phi_\theta(x)=F(x;\theta_F)$6; and progressive self-training alternates SGD updates of $\phi_\theta(x)=F(x;\theta_F)$7 and $\phi_\theta(x)=F(x;\theta_F)$8 with periodic label updates

$\phi_\theta(x)=F(x;\theta_F)$9

every $p_\theta(x)=C(\phi_\theta(x);\theta_C)$0 iterations until $p_\theta(x)=C(\phi_\theta(x);\theta_C)$1 or validation convergence (Kim et al., 2021).

SemiEvol follows a related propagate-and-select pattern, but in generative form. It first fine-tunes $p_\theta(x)=C(\phi_\theta(x);\theta_C)$2 on $p_\theta(x)=C(\phi_\theta(x);\theta_C)$3 to obtain $p_\theta(x)=C(\phi_\theta(x);\theta_C)$4, builds an embedding index over labeled tasks with $p_\theta(x)=C(\phi_\theta(x);\theta_C)$5 nearest-neighbor retrieval, queries $p_\theta(x)=C(\phi_\theta(x);\theta_C)$6 collaborating models for each unlabeled input, runs Self-Justify to produce $p_\theta(x)=C(\phi_\theta(x);\theta_C)$7, filters pseudo-responses by entropy percentile, and then fine-tunes $p_\theta(x)=C(\phi_\theta(x);\theta_C)$8 on $p_\theta(x)=C(\phi_\theta(x);\theta_C)$9 for 2 epochs to obtain $D_t^U$0. The paper also demonstrates up to four iterations, utilizing 94.75% of unlabeled data (Luo et al., 2024).

Active learning and incremental retraining introduce another recurrent pattern: query, pseudo-label, refine, retrain. In 3D medical segmentation, warm-start active learning trains a proxy model on HU-based pseudo-labels, selects the top-$D_t^U$1 most uncertain volumes for annotation at cold start, then, within each active iteration, trains a supervised model on the labeled pool and a semi-supervised model on the most certain unlabeled volumes with pseudo-label threshold $D_t^U$2 and consistency regularization (Nath et al., 2022). In multi-domain ASR, the incremental pipeline fine-tunes a seed model on $D_t^U$3, decodes the unlabeled pool once, filters it into subsets $D_t^U$4, and then repeatedly redecodes the accumulated buffer $D_t^U$5 with the previous model before fine-tuning the base model anew on $D_t^U$6 (Carofilis et al., 5 Jun 2025). In SAM-enabled medical segmentation, the three stages are Stitching, Fine-tuning, and Re-training: a 3D volume is stitched into a 2D mosaic, SAM ViT-B with LoRA rank $D_t^U$7 is fine-tuned on labeled mosaics, pseudo-labels are generated for unlabeled volumes, and a compact 3D segmenter is re-trained with method-specific semi-supervised losses (Li et al., 2024).

Partial-annotation NER implements yet another workflow. RoBERTa is first fitted on the partially labeled data, then GuidedBOND replaces teacher distributions with one-hot labels inside known entity spans,

$D_t^U$8

and k-fold Base Distribution Estimation builds soft targets for all tokens before a final RoBERTa model is fine-tuned on those base distributions (Scherbakov et al., 2022). The common structure across these examples is a staged alternation between reliable anchors, selective pseudo-supervision, and controlled model updates.

5. Empirical record across application domains

The empirical record is broad and highly domain-specific. In semi-supervised domain adaptation, the selective pseudo-labeling and progressive self-training method reports LSDAC average accuracy gains over MME of $D_t^U$9 vs $D_{\text{labeled}}=\{(t_i,y_i)\}$0 on AlexNet, $D_{\text{labeled}}=\{(t_i,y_i)\}$1 vs $D_{\text{labeled}}=\{(t_i,y_i)\}$2 on VGG-16, and $D_{\text{labeled}}=\{(t_i,y_i)\}$3 vs $D_{\text{labeled}}=\{(t_i,y_i)\}$4 on ResNet-34. On pseudo-label reliability, selection improves Clipart$D_{\text{labeled}}=\{(t_i,y_i)\}$5Sketch from $D_{\text{labeled}}=\{(t_i,y_i)\}$6 to $D_{\text{labeled}}=\{(t_i,y_i)\}$7 in 1-shot and from $D_{\text{labeled}}=\{(t_i,y_i)\}$8 to $D_{\text{labeled}}=\{(t_i,y_i)\}$9 in 3-shot; Painting$D_{\text{unlabeled}}=\{t_j\}$0Real improves from $D_{\text{unlabeled}}=\{t_j\}$1 to $D_{\text{unlabeled}}=\{t_j\}$2 in 1-shot and from $D_{\text{unlabeled}}=\{t_j\}$3 to $D_{\text{unlabeled}}=\{t_j\}$4 in 3-shot (Kim et al., 2021).

For LLM alignment, SemiEvol improves both closed and domain-specific evaluation. With GPT-4o-mini, MMLU moves from $D_{\text{unlabeled}}=\{t_j\}$5 to $D_{\text{unlabeled}}=\{t_j\}$6 to $D_{\text{unlabeled}}=\{t_j\}$7, MMLU-Pro from $D_{\text{unlabeled}}=\{t_j\}$8 to $D_{\text{unlabeled}}=\{t_j\}$9 to $D_t^L=\{(x_t^l,y_t^l)\}$00, ARC from $D_t^L=\{(x_t^l,y_t^l)\}$01 to $D_t^L=\{(x_t^l,y_t^l)\}$02 to $D_t^L=\{(x_t^l,y_t^l)\}$03, and ConvFinQA from $D_t^L=\{(x_t^l,y_t^l)\}$04 to $D_t^L=\{(x_t^l,y_t^l)\}$05 to $D_t^L=\{(x_t^l,y_t^l)\}$06. With Llama-3.1-8B, ConvFinQA moves from $D_t^L=\{(x_t^l,y_t^l)\}$07 to $D_t^L=\{(x_t^l,y_t^l)\}$08 to $D_t^L=\{(x_t^l,y_t^l)\}$09, corresponding to a $D_t^L=\{(x_t^l,y_t^l)\}$10 error reduction, and iterative rounds push MMLU-Pro beyond $D_t^L=\{(x_t^l,y_t^l)\}$11 while utilizing $D_t^L=\{(x_t^l,y_t^l)\}$12 of unlabeled data by iteration 4 (Luo et al., 2024). Selective Self-Rehearsal addresses a different empirical question—retention of generality after task adaptation—and reports that standard SFT can lead to an average performance drop of up to $D_t^L=\{(x_t^l,y_t^l)\}$13 on benchmarks such as MMLU and TruthfulQA, whereas SSR results in close to $D_t^L=\{(x_t^l,y_t^l)\}$14 drop on average (Gupta et al., 2024).

ASR and medical segmentation show similarly large gains when selection is strong. In multi-domain ASR, consensus-based filtering provides up to $D_t^L=\{(x_t^l,y_t^l)\}$15 relative improvement on Wow and $D_t^L=\{(x_t^l,y_t^l)\}$16 on Fisher over single-step fine-tuning with random selection; NER is the second-best filter and operates at lower computational cost (Carofilis et al., 5 Jun 2025). In the SAM-enabled semi-supervised 3D segmentation framework, Mean Teacher on the LA dataset improves from $D_t^L=\{(x_t^l,y_t^l)\}$17 to $D_t^L=\{(x_t^l,y_t^l)\}$18 Dice with only one labeled data, and the framework reports gains across LA, BraTS, BTCV, and MACT while discarding SAM at inference and retaining V-Net-scale deployment (Li et al., 2024).

Foundation-model SSL and parameter-efficient fine-tuning also show strong performance and efficiency. FineSSL reports new state of the art on multiple benchmark datasets and reduces the training cost by over six times; on CIFAR-10 it reaches $D_t^L=\{(x_t^l,y_t^l)\}$19 Top-1 for N1/N2/N4, on CIFAR-100 it reaches $D_t^L=\{(x_t^l,y_t^l)\}$20, and on ImageNet it reports $D_t^L=\{(x_t^l,y_t^l)\}$21 in the $D_t^L=\{(x_t^l,y_t^l)\}$22 labeled setting and $D_t^L=\{(x_t^l,y_t^l)\}$23 in the $D_t^L=\{(x_t^l,y_t^l)\}$24 labeled setting (Gan et al., 2024). LoFT, in a different but related regime, reports that fine-tuned foundation models can generate more reliable pseudo-labels, and that the method achieves superior performance even when utilizing only $D_t^L=\{(x_t^l,y_t^l)\}$25 of the unlabeled data compared with previous works (Chen et al., 12 Sep 2025). This suggests that selectivity can trade raw unlabeled volume for pseudo-label quality and still improve downstream accuracy.

6. Theory, failure modes, and open questions

The strongest formal analysis comes from the structural-causal model framework for semi-supervised domain adaptation. Under anticausal SCMs, the target-only minimax lower bound is

$D_t^L=\{(x_t^l,y_t^l)\}$26

so unlabeled target samples alone do not help estimate $D_t^L=\{(x_t^l,y_t^l)\}$27, and at least $D_t^L=\{(x_t^l,y_t^l)\}$28 labeled target samples are needed for nontrivial performance. By contrast, fine-tuning in low-dimensional subspaces identified from UDA starts reduces the problem from $D_t^L=\{(x_t^l,y_t^l)\}$29 parameters to $D_t^L=\{(x_t^l,y_t^l)\}$30 parameters, with FT-DIP, FT-OLS-Src, and FT-CIP achieving $D_t^L=\{(x_t^l,y_t^l)\}$31-type bounds under different shift assumptions. MASFT then trains multiple adaptive starts and selects among them with a small validation set, obtaining a model-selection guarantee of the form

$D_t^L=\{(x_t^l,y_t^l)\}$32

with $D_t^L=\{(x_t^l,y_t^l)\}$33 labeled validation examples (Ha et al., 19 Jul 2025). This suggests that selectivity can be viewed theoretically as a sample-complexity reduction mechanism, not only as a denoising heuristic.

Across applications, the reported limitations are also consistent. In selective pseudo-labeling for SSDA, $D_t^L=\{(x_t^l,y_t^l)\}$34 yields the best trade-off, while too small a ratio underutilizes data and ratios approaching $D_t^L=\{(x_t^l,y_t^l)\}$35 increase noise; features that are poorly clustered can require smaller $D_t^L=\{(x_t^l,y_t^l)\}$36, stronger augmentation, or consistency regularization (Kim et al., 2021). In SemiEvol, increasing the number of collaborators improves quality but increases compute, and too high a selection percentile admits noisy pseudo-labels (Luo et al., 2024). In volumetric medical segmentation, $D_t^L=\{(x_t^l,y_t^l)\}$37 and $D_t^L=\{(x_t^l,y_t^l)\}$38 control selection aggressiveness, small structures can have low margins even when correct, and early ASD foreground extraction may include false positives (Yang et al., 13 Sep 2025). In open-world LTSSL, aggressive OOD filtering can discard too many in-distribution samples, whereas weak filtering leaves OOD contamination in the pseudo-label pool (Chen et al., 12 Sep 2025). SSR adds a different limitation: if the judge misclassifies incorrect outputs as correct, pseudo-labels can introduce noise, and if the base model is poor on the task, $D_t^L=\{(x_t^l,y_t^l)\}$39 may be small, making SSR resemble ordinary SFT (Gupta et al., 2024).

A common misconception is that selective semi-supervised fine-tuning is synonymous with confidence thresholding on all unlabeled data. The literature does not support that reduction. Selectivity may be expressed through per-class quotas and feature anchors, entropy percentiles, judge-based gating, consensus or entity filters, branch detachment, gradient routing, grouped contrast, token masking, or validation-based choice among multiple fine-tuning starts (Kim et al., 2021, Luo et al., 2024, Gan et al., 2024, Lad et al., 2022, Ha et al., 19 Jul 2025). This suggests that the unifying principle is not any single pseudo-label rule, but the deliberate restriction of where semi-supervised signal is allowed to act.