Self-training Definition, uses & Variants

• Self-training is a semi-supervised learning approach, where a model trained on labeled data assigns pseudo-labels to unlabeled data, iteratively improving performance by incorporating these new labels. Self-training has been used in diverse applications across linguistic and non-linguistic fields including classification, sequence labeling, and speech recognition. Self-training models increasingly utilize teacher-student architectures, resilient-reinforced tuning methods, differentiable teacher strategies for enhanced performance.
• The paper explores numerous applications of self-training across different modalities and learning settings, highlighting significant improvements in these experiments. For example, self-training in natural language understanding tasks leads to better-quality work while reducing the same results as supervised-augmenting results, demonstrating enhanced performance over random-classifier models in applications ranging from speech recognition, to sequence tagging and entity alignment.
• The paper identifies several key challenges and methods to address confirmation bias and enhance robustness. Techniques including temperature calibration, doubly robust loss functions, and entropy-based selection have been proposed and validated across various settings, demonstrating improved performance over traditional semi-supervised methods.

Self-training is a semi-supervised learning strategy in which a model trained on a limited labeled dataset assigns pseudo-labels to selected unlabeled examples, incorporates those examples into subsequent training, and repeats the procedure. The central selection principle is usually confidence, margin, entropy, structural compatibility, or another estimate of pseudo-label reliability. Self-training can be implemented with a single classifier, teacher–student models, generative models, reinforcement-learning selectors, structured prediction systems, or differentiable teacher strategies. Its potential benefit is increased effective supervision; its principal risk is confirmation bias, in which incorrect pseudo-labels are treated as ground truth and propagated through later training iterations (Amini et al., 2022).

1. Definition, assumptions, and canonical procedure

Self-training begins with a labeled sample and a larger unlabeled sample. Let $S=\{(x_i,y_i)\}_{i=1}^{m}$ denote labeled observations and $U=\{x_i\}_{i=m+1}^{m+u}$ unlabeled observations, typically with $m\ll u$. A classifier produces a score $f(x,y)$ for each class and predicts

$h(x)=\arg\max_{y\in Y} f(x,y).$

At each iteration, the model is trained on the current labeled set, predicts labels for unlabeled examples, selects a subset judged sufficiently reliable, adds the selected pairs to the labeled data, and retrains. A generic pseudo-label for a selected example is

$\tilde y=h(x).$

The process terminates when no additional examples satisfy the selection rule, when the unlabeled pool is exhausted, or according to a predefined training or validation schedule. Pseudo-labeled examples may be assigned a lower loss weight than human-labeled examples through a coefficient such as $\gamma$, reflecting their greater uncertainty (Amini et al., 2022).

The principal statistical motivation is the smoothness or low-density-separation assumption: examples in high-density regions are likely to share labels, while decision boundaries should pass through low-density regions. High-margin or high-confidence predictions are therefore presumed less likely to be incorrect. For binary classification, an unsigned margin can be written as

$m_f(x)=|\rho_f(x,+1)|,$

where $\rho_f(x,y)$ is the difference between the score for class $y$ and the strongest competing score. In multiclass settings, selection may use maximum probability, the difference between the largest and second-largest scores, entropy, vote proportions, or class-dependent thresholds.

This assumption is conditional rather than universal. Unlabeled data are useful when their marginal distribution is compatible with the target labeling function. Under distribution shift, poor calibration, severe class imbalance, or a weak initial classifier, high confidence may not imply correctness. Consequently, self-training can improve a decision boundary, have little effect, or amplify systematic errors.

2. Selection mechanisms and pseudo-label quality

The selection rule determines the precision–coverage trade-off of self-training. Fixed thresholds are simple but sensitive: low thresholds admit more incorrect pseudo-labels, whereas high thresholds may select too few examples to provide useful additional supervision. Alternatives include proportion-based selection, dynamic class-specific thresholds, curriculum schedules, risk-bound threshold optimization, and entropy-based filtering (Amini et al., 2022).

Confidence-only selection is not always sufficient. In named-entity recognition, sentences containing no entities can receive high confidence because the model confidently assigns the outside tag to every token. A reinforcement-learning selector therefore uses sentence content, prediction marginals, tagger confidence, and hidden tagger features, learning from development-set performance rather than relying only on confidence. Its deep Q-network chooses between rejecting and accepting a pseudo-labeled sentence, with accepted examples rewarded according to the change in development performance (Chen et al., 2018).

Other domains require domain-specific selection signals. In optical character recognition, AT-ST ranks target-domain lines using the posterior probability of the decoded transcription under optical-only prefix search. This beam-normalized posterior measures the probability mass assigned to the selected sequence relative to competing hypotheses and outperformed several frame-level confidence measures for ranking pseudo-labeled handwritten and printed lines (Kišš et al., 2021). In entity alignment, pairwise similarity is augmented with graph-structural dependencies: a candidate mapping is favored when neighboring entity mappings support it. STEA converts neural similarities into probability distributions, applies dependency-aware inference, and selects mutually highest-probability mappings (Liu et al., 2022).

Pseudo-label representation is another design choice. Hard labels preserve only the argmax class, while soft labels retain the teacher’s probability distribution. Soft targets preserve uncertainty and class similarity information. In open-domain NLP self-training, continuous teacher probabilities trained with KL divergence improved the average performance over discrete labels by 0.9 percentage points (Du et al., 2020). Similar findings appear in enhanced NoisyStudent-style training, where soft pseudo-label loss outperformed hard loss on SVHN and CIFAR-10 and was selected as part of the Enhanced Self-Train configuration (Radhakrishnan et al., 2023).

Pseudo-label quality can also be controlled through sampling rather than explicit confidence masks. In semantic segmentation, centroid sampling selects crops around pseudo-labeled class or object centroids, ensuring that minority classes appear regularly and preventing frequent classes or systematic teacher errors from dominating training. This mechanism was used instead of a pixelwise confidence threshold in the reported segmentation framework (Zhu et al., 2020).

3. Teacher–student architectures and training variants

In the teacher–student formulation, a teacher generates pseudo-labels and a student is trained on human and pseudo-labeled data. The student may replace the teacher in the next iteration, as in iterative self-training for task-oriented dialog, or the teacher may be an exponential moving average of student parameters. In dialog systems, the teacher selects the top-$U=\{x_i\}_{i=m+1}^{m+u}$0 most confident unlabeled examples, the student is reinitialized and trained on the expanded dataset, and the resulting student becomes the next teacher (Mi et al., 2021).

The student can be trained jointly on clean and pseudo-labeled data, initialized from the teacher, trained from scratch, or fine-tuned from previous iterations. In semantic segmentation, joint training on real and pseudo-labeled data outperformed pseudo-label pretraining followed by fine-tuning. In enhanced NoisyStudent training, fine-tuned student initialization improved performance over fresh student training, while SplitBatch and MixedLoss controlled the ratio and loss contribution of clean and pseudo-labeled examples (Zhu et al., 2020, Radhakrishnan et al., 2023).

Self-supervision can improve the first pseudo-labeling iteration. In image classification, STSSC combines classification and geometric-transformation prediction only during the first self-training iteration, then uses ordinary self-training. The transformations are rotations and horizontal or vertical flips. This strategy generally outperformed self-supervised pretraining alone and continuous combined training, while requiring less computation than maintaining the auxiliary loss throughout all iterations (Sahito et al., 2021).

Teacher and student need not differ in architecture. DRIFT, or differentiable self-training, treats the student as the leader and the teacher as a differentiable follower in a Stackelberg game. It replaces nondifferentiable hard pseudo-labels with temperature-controlled soft labels and assigns entropy-based differentiable sample weights. The student gradient includes both the direct student-loss term and the derivative through the teacher strategy. With an exponential moving average teacher,

$U=\{x_i\}_{i=m+1}^{m+u}$1

This formulation explicitly accounts for how student updates alter future pseudo-labels and weights (Zuo et al., 2021).

A different architectural use of self-training appears in STICL, which transfers the demonstration-induced behavior of an in-context-learning teacher into a query-only student equipped with a LoRA adapter. The teacher receives demonstrations and a query; the student receives only the query and is trained against teacher outputs. The adapter stores the latent shift associated with the demonstrations, reducing inference-time context length and sensitivity to demonstration ordering (Jukić et al., 2024).

4. Applications across modalities and learning settings

Self-training has been applied to classification, sequence labeling, parsing, speech, vision, structured prediction, continual learning, and reasoning.

In natural language understanding, SentAugment retrieves task-relevant sentences from a CommonCrawl-derived bank using task-specific sentence-embedding queries. A RoBERTa-Large teacher pseudo-labels the retrieved sentences, and a student is trained on confidence-filtered, class-balanced examples. On six downstream tasks, self-training improved the average score from 87.4 to 88.6, while continued masked-language-model pretraining on the same retrieved text reduced the average to 86.2, indicating that task-conditioned pseudo-labeling contributed information beyond generic pretraining (Du et al., 2020).

In cross-lingual reading comprehension, a multilingual BERT model is first trained on a labeled source-language extractive-QA dataset, then used to pseudo-label unlabeled target-language questions and contexts. Candidate answer spans are generated by beam search and retained when the sum of start and end probabilities exceeds 0.7. Across twelve source-target directions involving English, French, Korean, and Traditional Chinese, self-training improved both exact match and F1 (Huang et al., 2021).

For conversational speech, a self-attentive constituency parser trained on gold Switchboard trees pseudo-labels unlabeled Fisher transcripts. A student is retrained on a 40% silver-Fisher and 60% gold-Switchboard mixture. Self-training improved EDITED-span F-score from 89.2 to 90.7 and EDITED-word F-score from 90.9 to 92.4; the largest error-analysis gains were for corrections and restarts (Lou et al., 2020).

In speech translation, cascade systems or end-to-end speech-translation models generate translations for unlabeled audio. Training a larger student on up to 35,217 hours of LibriLight audio, followed by fine-tuning on trusted data, improved MuST-C English–French by 8.3 BLEU and English–German by 5.7 BLEU over a strong semi-supervised baseline. Pseudo-labeling outperformed ASR encoder pretraining because it supervises both the speech encoder and translation decoder (Pino et al., 2020).

In semantic segmentation, a teacher trained on human pixel annotations generates dense hard pseudo-labels for Cityscapes coarse images and Mapillary images. Joint training with centroid-controlled sampling yielded 80.0% mIoU in the principal Cityscapes ablation and a reported 82.7% test mIoU using Cityscapes fine labels plus pseudo-labels from unlabeled images. The approach also improved cross-domain transfer to CamVid, KITTI, and Mapillary (Zhu et al., 2020).

In class-incremental semantic segmentation, self-training acts as rehearsal without retaining previous labeled data. A temporary model learns current classes, while the old model preserves previous classes. Their predictions on auxiliary unlabeled images are fused, conflicts are resolved using maximum class probabilities, and self-entropy maximization reduces overconfidence. The method obtained 61.2% mIoU versus 37.9% for MiB in the Pascal-VOC 15-1 disjoint protocol and 63.6% versus 29.7% in the overlapped protocol (Yu et al., 2020).

For automatic speech recognition, NSTI performs noisy student–teacher adaptation on the recording that is about to be transcribed. The recording is segmented, pseudo-labeled, augmented with frequency masking, and used for adaptation before final decoding. On CHiME-6, test WER decreased from 86.5 to 59.4, corresponding to a reported 31.3% relative WER reduction; the abstract reports gains up to 32.2%. The method is offline and exploits correlations across utterances within the same recording (Flynn et al., 2024).

5. Confirmation bias, robustness, and failure modes

Confirmation bias is the central failure mechanism of iterative self-training. An incorrect teacher prediction becomes a student target; the student then becomes a more confident future teacher, reinforcing the same error. High-confidence errors, class imbalance, pseudo-labeled data dominating minibatches, and poor initial models intensify this process (Radhakrishnan et al., 2023).

Several methods explicitly target this problem. Temperature calibration and entropy-based selection replace raw maximum-softmax thresholds in Enhanced Self-Train. Weighted SplitBatch sampling combines inverse class frequency with within-class confidence, while MixedLoss ensures that clean labeled examples retain substantial influence. On the larger-model evaluation, Enhanced Self-Train improved over a reimplemented NoisyStudent baseline from 89.15% to 94.21% on CIFAR-10 and from 70.53% to 76.42% on CIFAR-100 (Radhakrishnan et al., 2023).

Doubly Robust Self-Training addresses pseudo-label bias at the objective level. Its loss is

$U=\{x_i\}_{i=m+1}^{m+u}$2

The teacher provides an estimate using all examples, while labeled data estimate and correct the teacher’s error. If pseudo-labels are completely incorrect, the method asymptotically reduces to labeled-only training; if they are completely accurate, it uses the full pseudo-labeled and labeled sample. On ImageNet100 and nuScenes, the method generally improved over naive self-training, particularly in low-label regimes (Zhu et al., 2023).

Some methods exploit structure rather than merely confidence. STEA uses neighborhood dependencies to suppress false-positive entity mappings without sacrificing true-positive recall. In entity alignment with 1% annotation, it reached Hit@1 values of 0.723 on zh_en, 0.471 on ja_en, 0.584 on fr_en, 0.866 on dbp_wd, and 0.718 on dbp_yg, substantially exceeding similarity-threshold and mutual-nearest-neighbor baselines (Liu et al., 2022).

Differentiable teacher methods address instability caused by abrupt teacher changes and hard $U=\{x_i\}_{i=m+1}^{m+u}$3 decisions. DRIFT’s soft pseudo-labels and entropy weights vary smoothly with teacher parameters, and its total derivative anticipates the effect of student updates on future supervision. It outperformed conventional self-training in semi-supervised and weakly supervised classification, named-entity recognition, and graph node classification (Zuo et al., 2021).

Self-training may also fail because the unlabeled distribution is incompatible with the labeled task. Open-set unlabeled data can contain unseen classes, causing forced known-class pseudo-labels. Enhanced Self-Train uses contrastive features, class prototypes, and Beta-distribution distance models to filter examples that appear far from every known class prototype (Radhakrishnan et al., 2023). In continual segmentation, unrelated but diverse auxiliary data were useful, whereas narrowly focused CUB and Flowers data performed poorly relative to COCO and ImageNet (Yu et al., 2020).

6. Advanced, test-time, and self-rewarded self-training

Self-training is increasingly used beyond conventional offline semi-supervised learning. In test-time speech recognition, NSTI adapts separately to each recording and does not retain the adapted model for later recordings. It can outperform adaptation using a much larger separate corpus because the current recording is more representative of its own speakers, microphones, acoustics, topics, and vocabulary. The trade-off is increased inference-time computation and the requirement that the complete recording be available before final transcription (Flynn et al., 2024).

STICL similarly transfers context-dependent behavior into parameters. Its teacher is a full-prompt ICL model, while its student is a query-only adapter model. STICL-Shuffle improved standard ICL on RTE from 75.1 to 86.0, on SST from 93.5 to 96.1, and on MMLU mathematics from 43.5 to 49.5 in the reported Llama 3 experiments. It also reduced sensitivity to demonstration ordering (Jukić et al., 2024).

Self-training can be combined with reinforcement learning when no external labels or verifier are available. Self-Rewarded Training uses majority agreement among independently sampled mathematical solutions as a proxy reward. A response receives reward 1 when its parsed answer matches the majority answer and 0 otherwise. Early training on MATH, AIME, and DAPO produced improvements comparable to reinforcement learning with gold answers, with peak performance reaching roughly a 100% relative improvement over the base model across reported configurations (Shafayat et al., 27 May 2025).

The self-rewarded setting exposes a fundamental limitation: agreement is not correctness. On DAPO, SRT initially improved but later collapsed. Self-reward continued increasing while held-out accuracy declined, KL divergence from the reference model rose, and outputs became prompt-independent and consistently wrong. The reward admits degenerate solutions that maximize consistency without solving the task. Early stopping, frozen pseudo-labels, and training on an easy subset delayed or reduced collapse, but prolonged unconstrained online optimization remained vulnerable to reward hacking (Shafayat et al., 27 May 2025).

7. Evaluation, theoretical status, and research directions

Empirical evaluation of self-training typically compares a supervised baseline, a confidence- or structure-based self-training method, and variants that alter pseudo-label representation, sampling, augmentation, teacher updating, or iteration count. Reported outcomes include classification accuracy, F1, mIoU, BLEU, WER, CER, Hit@1, MRR, exact match, and task-specific detection metrics. Results are often strongest when the initial teacher is sufficiently reliable, the unlabeled data are compatible with the target task, and pseudo-label influence is explicitly controlled.

Theoretical guarantees are conditional. Existing analyses cover restricted classifiers, data distributions, noise models, and optimization procedures. The survey reports results for majority-vote bounds, expansion conditions, linear models, two-layer neural networks, and Massart-noise settings, but these do not establish universal guarantees for deep self-training under arbitrary distribution shift (Amini et al., 2022). Doubly Robust Self-Training provides a more general bias-correction perspective under matching covariate distributions and regularity assumptions, while an importance-weighted extension addresses covariate-distribution mismatch (Zhu et al., 2023).

Several recurring design principles emerge from the research:

1. Teacher quality matters: pseudo-labels generated by a weak model can degrade performance, even when large quantities of unlabeled data are available.
2. Selection quality matters more than quantity: indiscriminately adding more examples can be harmful; confidence, entropy, calibration, structural compatibility, and diversity all affect utility.
3. Soft supervision is often safer: probability distributions preserve uncertainty and reduce the damage caused by incorrect hard labels.
4. Clean labels must remain influential: joint training, controlled batch composition, loss correction, or explicit reweighting prevent pseudo-labels from overwhelming trusted supervision.
5. Iteration is not automatically beneficial: later rounds may yield diminishing returns, saturation, or collapse. Validation-based stopping, frozen teachers, or one-pass procedures can be preferable.
6. Domain structure can improve selection: graph dependencies, target-domain appearance, sequence-level alternatives, recording-level correlations, and class prototypes provide information that generic confidence does not capture.
7. Self-training and pretraining are complementary: contextualized representations, pretrained vision models, task-oriented dialog encoders, and multilingual encoders can provide strong teachers, while pseudo-labeling supplies task- or domain-specific supervision.
8. Unlabeled data can be useful without being strictly in-domain: retrieved web sentences, diverse auxiliary images, and current test recordings can support self-training when selection or adaptation mechanisms make their information relevant.

The main unresolved issues concern calibrated uncertainty, robustness under severe distribution shift, computational efficiency, global modeling of unlabeled pools, open-set detection, theoretical analysis of confirmation bias, and safe optimization when the supervisory signal is generated by the model itself. The literature supports self-training as a flexible mechanism for converting unlabeled observations into task-specific supervision, but its reliability depends on preventing pseudo-label errors from becoming the dominant source of learning.