---
title: Selective Semi-supervised Fine-tuning
url: https://www.emergentmind.com/topics/selective-semi-supervised-fine-tuning
type: topic
---

# Selective Semi-supervised Fine-tuning

Selective semi-supervised fine-tuning denotes a family of adaptation procedures in which a pretrained or warm-started model is updated with a small labeled set and a larger unlabeled pool, while the optimization is made selective with respect to samples, pseudo-labels, parameters, latent subspaces, or even token positions. In the cited literature, this label covers semi-supervised domain adaptation, foundation-model adaptation, large language model alignment, ASR, medical segmentation, NER with partial annotation, and theoretically grounded target-domain fine-tuning from multiple adaptive starts [2104.00319] [2410.14745] [2507.14661].

## 1. Problem formulations and scope

A canonical formulation appears in semi-supervised domain adaptation, where the source domain is \(D_s=\{(x_s,y_s)\}\), the target domain has a few labeled samples \(D_t^L=\{(x_t^l,y_t^l)\}\), typically 1-shot or 3-shot per class, and a larger unlabeled set \(D_t^U=\{x_t^u\}\). The model is decomposed into a feature extractor \(F(\cdot;\theta_F)\) and classifier \(C(\cdot;\theta_C)\), with feature \(\phi_\theta(x)=F(x;\theta_F)\) and prediction \(p_\theta(x)=C(\phi_\theta(x);\theta_C)\). The target-oriented model is then fine-tuned by leveraging the small labeled target set together with selectively chosen pseudo labels for \(D_t^U\) [2104.00319].

In large language model alignment, the same structure is written as a labeled set \(D_{\text{labeled}}=\{(t_i,y_i)\}\) and an unlabeled set \(D_{\text{unlabeled}}=\{t_j\}\). The objective is to propagate target-task knowledge from labeled data to unlabeled data, generate and select high-quality pseudo-responses, and fine-tune the LLM to yield an evolved model adapted to the target domain [2410.14745]. In long-tailed semi-supervised learning with foundation models, the labeled set \(\mathcal{D}_l\) is imbalanced, the unlabeled set \(\mathcal{D}_u\) may not match the labeled distribution, and, in open-world settings, \(\mathcal{D}_u\) may also contain out-of-distribution samples [2509.09926].

The same logic extends beyond ordinary labeled/unlabeled splits. In partially annotated NER corpora, token positions are partitioned into trusted indices \(L\) and unreliable or missing positions \(U\); many true entity tokens are marked as “O” despite being entity tokens. The fine-tuning problem then becomes selective both over tokens and over supervision type, because known spans are enforced while the remaining positions are trained through soft targets rather than hard negatives [2211.14360]. This suggests that selective semi-supervised fine-tuning is better understood as a family of constrained or filtered fine-tuning regimes than as a single algorithmic template.

## 2. Data selection and pseudo-label curation

The most direct form of selectivity is explicit sample selection for pseudo-labeling. In semi-supervised domain adaptation, unlabeled target samples are first assigned soft pseudo labels \(\widetilde y(x)=p_{\hat\theta}(x)\) and hard pseudo labels \(\hat y(x)=\arg\max_k p_{\hat\theta}(y=k\mid x)\). Selection is then driven not by a confidence threshold but by feature-space distance to labeled target samples of the same predicted class:
$$
d(x;c)=\frac{1}{n_t'(c)}\sum_{(x_t^l,y_t^l)\in I_c}\|\phi_{\hat\theta}(x_t^l)-\phi_{\hat\theta}(x)\|_1.
$$
For each class \(c\), unlabeled targets with \(\hat y(x)=c\) are sorted by ascending \(d(x;c)\), and the top \(n_u'(c)=\left\lceil r_u\cdot |D_t^U|/K\right\rceil\) are selected, with default \(r_u=0.2\). Because \(n_u'(c)\) is fixed per class, the selected pseudo-labeled set is balanced across classes [2104.00319].

Large language model methods often select by response quality rather than feature geometry. SemiEvol generates multiple candidate responses, aggregates them with a Self-Justify step, computes response entropy
$$
H(\tilde y_j)=-\frac{1}{L_j}\sum_{k=1}^{L_j}\log P(r_j^k\mid t_j,r_j^{<k}),
$$
and keeps only pseudo-responses whose entropy is below a dynamic threshold \(\tau=\mathrm{Percentile}_\theta(\{H(\tilde y_j)\})\) with \(\theta=50\%\) [2410.14745]. Selective Self-Rehearsal uses an LLM-as-a-judge instead: if \(J(x_i,y_i^{\text{model}})=1\), the model-generated response is used as the training target; otherwise the gold response is retained. The loss becomes
$$
L(D)=-\sum_{i\in S_{\text{correct}}}\log Pr_\theta(y_i^{\text{model}}\mid x_i)-\sum_{j\in S_{\text{gold}}}\log Pr_\theta(y_j^{\text{gold}}\mid x_j),
$$
thereby turning pseudo-labeling into a judge-gated substitution problem rather than an unlabeled-data expansion problem [2409.04787].

Speech and 3D medical imaging use other selection rules. In multi-domain ASR, pseudo-labeled utterances are filtered either by multi-model consensus, using average pairwise CER
$$
\bar{\mathrm{CER}}=\frac{\mathrm{CER}(T_1,T_2)+\mathrm{CER}(T_1,T_3)+\mathrm{CER}(T_2,T_3)}{3},
$$
with retain-if \(\bar{\mathrm{CER}}<0.05\), or by NER, retaining segments with at least one detected named entity [2506.04981]. In Active Source-Free Domain Adaptation for volumetric medical segmentation, reliability is defined as
$$
r(x)=C(x)\cdot [1-D(x)],
$$
where \(C(x)\) is a foreground-aware confidence score and \(D(x)\) is cosine distance to labeled anchors; the top \(\mathcal N_{SU}\) volumes are then selected, and low-confidence voxels within them are masked with \(M_{pl}(v)=1[P_{\max}(v)-P_{\max}^{(2)}(v)\ge \delta_{pl}]\) [2509.10784]. In open-world long-tailed SSL, LoFT-OW first prefilters unlabeled data by zero-shot confidence with \(t_{\mathrm{HC}}=0.95\), then applies MSP-based filtering and a confidence split between hard pseudo-labels and soft consistency supervision [2509.09926].

## 3. Selective objectives, parameters, and representations

Selectivity is not limited to sample choice. In several methods it is embedded directly into the objective or into gradient routing. In content-style decomposition for vision foundation models, the backbone latent \(\tilde y_x\) is decomposed into content \(c\) and style \(s\), the supervised loss is
$$
L_{\text{sup}}=\mathbb E_{(x,y)\in D_L}[-\log p_{\theta c}(y\mid \tilde y_x)],
$$
and the semi-supervised part is
$$
L_{\text{semi}}=\mathbb E_{x\in D_U}[\lambda_c\tilde D_c+\lambda_s\tilde D_s+\lambda_y\tilde D_y+\lambda_{rec}L_{rec}(\tilde y_x,\hat y_x)].
$$
Crucially, the backbone is updated only by supervised gradients, \(\partial L/\partial \phi_x=\partial L_{\text{sup}}/\partial \phi_x\), while \(\partial L_{\text{semi}}/\partial \phi_x=0\). This is selective semi-supervised fine-tuning at the parameter level: the backbone receives only supervised gradients; the content, style, decoder, and discriminator heads receive unlabeled regularization [2410.02069].

FineSSL makes selectivity class-aware and branch-aware. Balanced Margin Softmax computes a class-dependent margin from unlabeled learning pace,
$$
\sigma_t(k)=\sum_{j=1}^{M}\mathbb I(\max(\mathbf q_j)\ge \zeta)\cdot \mathbb I(\arg\max(\mathbf q_j)=k),\quad
\beta_t(k)=\frac{\sigma_t(k)}{\max_c\sigma_t(c)},\quad
\Delta_t(k)=1-\beta_t(k),
$$
and uses
$$
\mathcal H_m(y,\mathbf z)=-\log\frac{\exp(z_y)}{\exp(z_y)+\sum_{k\ne y}\exp(z_k+\alpha_t\Delta_t(y))}.
$$
Unlabeled samples are not hard-dropped in the main branch; instead they are weighted by an auxiliary classifier,
$$
\psi(u_j)=\gamma\cdot \max_c \mathrm{Softmax}(f_{\mathrm{aux}}(\omega(u_j)))_c,
$$
while Decoupled Label Smoothing applies smoothing only in the detached auxiliary branch through
$$
\tilde{\mathbf q}_j=(1-\lambda)\mathbf e_{\hat q_j}+\frac{\lambda}{C}\mathbf 1.
$$
This produces selective pressure across classes and across branches rather than only across samples [2405.11756].

Other methods make the objective selective through grouped contrast or through trainable-head restriction. Self-Tuning keeps a pretrained backbone, optimizes supervised cross-entropy on labeled data, and replaces unsupervised CE with Pseudo Group Contrast, where an unlabeled sample with pseudo-label \(\hat y\) contrasts its query against \(\{k_{\text{self}}\}\cup Q_{\hat y}\) and against all other class queues. The resulting grouped positives make low-quality pseudo-labels contribute small gradients without any hard confidence threshold [2102.12903]. TwinTURBO freezes the foundation backbone and trains only a shallow transformer, projector, predictor, and critic, combining a task-space mutual-information bound with a latent-space InfoNCE-style loss; the total objective is
$$
\mathcal L_{\text{total}}=\mathcal L_{\text{task}}+\lambda_{\text{latent}}\mathcal L^{\text{latent}},
$$
with \(\mathcal L_{\text{task}}\) itself combining conditional and marginal cross-entropy and a JSD-based critic [2503.07851]. This suggests that selective semi-supervised fine-tuning often refers as much to which parameters are allowed to move and which losses are allowed to reach them as to which samples are admitted.

A related pre-fine-tuning form appears in task-specific masking for language models. A word score
$$
s(w)=\frac{|u^\top x_w+b|}{\|u\|}
$$
is computed from seed word lists via a linear SVM in Word2Vec space, mapped to a masking probability \(p_{\text{mask}}(w)\) through step, linear, or exponential functions, and used in a short MLM bridge stage before standard supervised fine-tuning [2211.13815]. Here the selective unit is not the sample or the parameter, but the token position.

## 4. Algorithmic workflows

Many selective semi-supervised fine-tuning pipelines share a multi-stage schedule. In semi-supervised domain adaptation, the baseline warm-up uses Minimax Entropy to obtain \((\hat\theta_F,\hat\theta_C)\); a selected set \(S\subseteq D_t^U\) is then built by feature-space distance; soft pseudo labels are initialized as \(\widetilde y(x)=p_{\hat\theta}(x)\); and progressive self-training alternates SGD updates of \(\theta_F\) and \(\theta_C\) with periodic label updates
$$
\widetilde y(x)\leftarrow m\cdot \widetilde y(x)+(1-m)\cdot p_\theta(x),\quad m=0.9,
$$
every \(t_{\text{val}}=500\) iterations until \(t_{\max}=50{,}000\) or validation convergence [2104.00319].

SemiEvol follows a related propagate-and-select pattern, but in generative form. It first fine-tunes \(M_{\text{base}}\) on \(D_{\text{labeled}}\) to obtain \(M_{\text{warm}}\), builds an embedding index over labeled tasks with \(k=3\) nearest-neighbor retrieval, queries \(n=4\) collaborating models for each unlabeled input, runs Self-Justify to produce \(\tilde y_j\), filters pseudo-responses by entropy percentile, and then fine-tunes \(M_{\text{warm}}\) on \(D_{\text{selected}}\) for 2 epochs to obtain \(M_{\text{evol}}\). The paper also demonstrates up to four iterations, utilizing 94.75% of unlabeled data [2410.14745].

Active learning and incremental retraining introduce another recurrent pattern: query, pseudo-label, refine, retrain. In 3D medical segmentation, warm-start active learning trains a proxy model on HU-based pseudo-labels, selects the top-\(k\) most uncertain volumes for annotation at cold start, then, within each active iteration, trains a supervised model on the labeled pool and a semi-supervised model on the most certain unlabeled volumes with pseudo-label threshold \(\tau=0.9\) and consistency regularization [2209.06285]. In multi-domain ASR, the incremental pipeline fine-tunes a seed model on \(S_{\text{core}}\cup S_{\text{aux}}\), decodes the unlabeled pool once, filters it into subsets \(\{U_1,\dots,U_K\}\), and then repeatedly redecodes the accumulated buffer \(U_i'\) with the previous model before fine-tuning the base model anew on \(S_{\text{core}}\cup U_i'\) [2506.04981]. In SAM-enabled medical segmentation, the three stages are Stitching, Fine-tuning, and Re-training: a 3D volume is stitched into a 2D mosaic, SAM ViT-B with LoRA rank \(r=4\) is fine-tuned on labeled mosaics, pseudo-labels are generated for unlabeled volumes, and a compact 3D segmenter is re-trained with method-specific semi-supervised losses [2403.11229].

Partial-annotation NER implements yet another workflow. RoBERTa is first fitted on the partially labeled data, then GuidedBOND replaces teacher distributions with one-hot labels inside known entity spans,
$$
\hat p_j^k=
\begin{cases}
\mathbb 1[j=l_k], & \text{if } \exists (m,n)\in E \text{ with } m\le k<n,\\
p_j^k, & \text{otherwise},
\end{cases}
$$
and k-fold Base Distribution Estimation builds soft targets for all tokens before a final RoBERTa model is fine-tuned on those base distributions [2211.14360]. The common structure across these examples is a staged alternation between reliable anchors, selective pseudo-supervision, and controlled model updates.

## 5. Empirical record across application domains

The empirical record is broad and highly domain-specific. In semi-supervised domain adaptation, the selective pseudo-labeling and progressive self-training method reports LSDAC average accuracy gains over MME of \(47.6\) vs \(44.2\) on AlexNet, \(61.5\) vs \(59.2\) on VGG-16, and \(69.0\) vs \(66.4\) on ResNet-34. On pseudo-label reliability, selection improves Clipart\(\to\)Sketch from \(35.2\%\) to \(61.6\%\) in 1-shot and from \(41.0\%\) to \(64.8\%\) in 3-shot; Painting\(\to\)Real improves from \(57.7\%\) to \(83.8\%\) in 1-shot and from \(60.7\%\) to \(85.8\%\) in 3-shot [2104.00319].

For LLM alignment, SemiEvol improves both closed and domain-specific evaluation. With GPT-4o-mini, MMLU moves from \(77.4\) to \(77.8\) to \(79.9\), MMLU-Pro from \(57.8\) to \(58.8\) to \(60.8\), ARC from \(91.5\) to \(90.3\) to \(92.7\), and ConvFinQA from \(63.9\) to \(88.8\) to \(89.2\). With Llama-3.1-8B, ConvFinQA moves from \(51.1\) to \(81.3\) to \(83.6\), corresponding to a \(66.5\%\) error reduction, and iterative rounds push MMLU-Pro beyond \(55\%\) while utilizing \(94.75\%\) of unlabeled data by iteration 4 [2410.14745]. Selective Self-Rehearsal addresses a different empirical question—retention of generality after task adaptation—and reports that standard SFT can lead to an average performance drop of up to \(16.7\%\) on benchmarks such as MMLU and TruthfulQA, whereas SSR results in close to \(2\%\) drop on average [2409.04787].

ASR and medical segmentation show similarly large gains when selection is strong. In multi-domain ASR, consensus-based filtering provides up to \(22.3\%\) relative improvement on Wow and \(24.8\%\) on Fisher over single-step fine-tuning with random selection; NER is the second-best filter and operates at lower computational cost [2506.04981]. In the SAM-enabled semi-supervised 3D segmentation framework, Mean Teacher on the LA dataset improves from \(29.68\%\) to \(74.40\%\) Dice with only one labeled data, and the framework reports gains across LA, BraTS, BTCV, and MACT while discarding SAM at inference and retaining V-Net-scale deployment [2403.11229].

Foundation-model SSL and parameter-efficient fine-tuning also show strong performance and efficiency. FineSSL reports new state of the art on multiple benchmark datasets and reduces the training cost by over six times; on CIFAR-10 it reaches \(96.15/96.87/97.39\%\) Top-1 for N1/N2/N4, on CIFAR-100 it reaches \(80.44/84.51/86.66\%\), and on ImageNet it reports \(74.22\%\) in the \(1\%\) labeled setting and \(79.21\%\) in the \(10\%\) labeled setting [2405.11756]. LoFT, in a different but related regime, reports that fine-tuned foundation models can generate more reliable pseudo-labels, and that the method achieves superior performance even when utilizing only \(1\%\) of the unlabeled data compared with previous works [2509.09926]. This suggests that selectivity can trade raw unlabeled volume for pseudo-label quality and still improve downstream accuracy.

## 6. Theory, failure modes, and open questions

The strongest formal analysis comes from the structural-causal model framework for semi-supervised domain adaptation. Under anticausal SCMs, the target-only minimax lower bound is
$$
\min_{\hat f}\sup_{\mathcal P}\mathbb E[\delta R_T(\hat f)]\ge c v_Y^2\min\{d/n_\ell,1\},
$$
so unlabeled target samples alone do not help estimate \(\beta_T^*\), and at least \(O(d)\) labeled target samples are needed for nontrivial performance. By contrast, fine-tuning in low-dimensional subspaces identified from UDA starts reduces the problem from \(d\) parameters to \(r\) parameters, with FT-DIP, FT-OLS-Src, and FT-CIP achieving \(r/n_\ell\)-type bounds under different shift assumptions. MASFT then trains multiple adaptive starts and selects among them with a small validation set, obtaining a model-selection guarantee of the form
$$
\delta R_T(\hat f_{\text{sel}})\le \min_{j\in[L]}\delta R_T(\hat f_j)+c'\sigma_\ell^2\sqrt{\log(2L/\delta)/n_v},
$$
with \(n_v\gtrsim \log M\) labeled validation examples [2507.14661]. This suggests that selectivity can be viewed theoretically as a sample-complexity reduction mechanism, not only as a denoising heuristic.

Across applications, the reported limitations are also consistent. In selective pseudo-labeling for SSDA, \(r_u\approx 0.2\) yields the best trade-off, while too small a ratio underutilizes data and ratios approaching \(1.0\) increase noise; features that are poorly clustered can require smaller \(r_u\), stronger augmentation, or consistency regularization [2104.00319]. In SemiEvol, increasing the number of collaborators improves quality but increases compute, and too high a selection percentile admits noisy pseudo-labels [2410.14745]. In volumetric medical segmentation, \(\delta_{pl}\) and \(\tau_c\) control selection aggressiveness, small structures can have low margins even when correct, and early ASD foreground extraction may include false positives [2509.10784]. In open-world LTSSL, aggressive OOD filtering can discard too many in-distribution samples, whereas weak filtering leaves OOD contamination in the pseudo-label pool [2509.09926]. SSR adds a different limitation: if the judge misclassifies incorrect outputs as correct, pseudo-labels can introduce noise, and if the base model is poor on the task, \(S_{\text{correct}}\) may be small, making SSR resemble ordinary SFT [2409.04787].

A common misconception is that selective semi-supervised fine-tuning is synonymous with confidence thresholding on all unlabeled data. The literature does not support that reduction. Selectivity may be expressed through per-class quotas and feature anchors, entropy percentiles, judge-based gating, consensus or entity filters, branch detachment, gradient routing, grouped contrast, token masking, or validation-based choice among multiple fine-tuning starts [2104.00319] [2410.14745] [2405.11756] [2211.13815] [2507.14661]. This suggests that the unifying principle is not any single pseudo-label rule, but the deliberate restriction of where semi-supervised signal is allowed to act.

Source: https://www.emergentmind.com/topics/selective-semi-supervised-fine-tuning