Iterative Pseudo Labeling Techniques

Updated 3 July 2026

Iterative pseudo labeling is a semi-supervised learning method that alternates between generating and refining pseudo-labels to effectively utilize unlabeled data.
The technique employs filtering methods like confidence thresholds and committee voting to minimize error propagation and confirmation bias during retraining.
It has demonstrated significant performance improvements across fields such as speech recognition, object detection, medical imaging, and low-resource NLP.

Iterative pseudo labeling is a foundational paradigm in semi-supervised learning, self-training, and domain adaptation that aims to exploit unlabeled data by repeatedly generating, filtering, and leveraging machine-generated surrogate labels (“pseudo-labels”) in cycles of model refinement. This process, characterized by multiple rounds of pseudo-label generation followed by model retraining or adaptation, is essential in settings where labels are scarce, expensive, or costly to acquire. Iterative pseudo labeling has been shown to yield significant performance gains across diverse domains including speech recognition, object detection, semantic segmentation, knowledge graph alignment, low-resource NLP classification, and medical imaging.

1. The Iterative Pseudo Labeling Principle

At its core, iterative pseudo labeling alternates between the following steps:

Inference: Use the current model to predict (with or without confidence thresholds) pseudo-labels on the unlabeled data pool.
Filtering/Selection: Optionally filter these labels based on confidence, margin, or other uncertainty measures to retain high-quality pseudo-labels.
Fusion: Merge selected pseudo-labeled examples with existing human-labeled data (support set, seed set, or labeled pool).
Retraining: Retrain or fine-tune the model on the expanded data, potentially weighting pseudo-label and supervised loss terms differently (e.g., via a coefficient schedule).
Iteration: Repeat the loop, dynamically updating pseudo-labels as the model evolves, until a stopping criterion (e.g., validation metric plateau, maximum iterations, convergence) is met.

This cycle enables both gradual quality improvement of the pseudo-labels (as the model becomes more confident and accurate) and progressive domain/representation adaptation, particularly in low-resource or cross-domain regimes.

In practice, methods may use hard (argmax class) or soft (probability vector) pseudo-labels, perform filtering via confidence or more sophisticated scoring (KL divergence, committee voting), enforce curriculum constraints, or include additional domain-specific refinements (Liu et al., 28 May 2026, Silovsky et al., 2023, Liu et al., 21 Feb 2026, Xu et al., 2020, Asano et al., 18 Feb 2025, Benato et al., 2021, Lazarou et al., 2020, Zhang et al., 2022, Zhao et al., 1 Apr 2026, Jin et al., 6 Aug 2025, Higuchi et al., 2021, Cascante-Bonilla et al., 2020).

2. Representative Methodologies and Algorithmic Variants

The iterative pseudo labeling loop admits multiple methodological instantiations:

a) Self-Training with Confidence Scheduling

Selects pseudo-labeled examples above a confidence threshold or percentile (possibly progressively relaxed). For example, in Curriculum Labeling, the percentile-threshold (“curriculum”) decays across iterations and models are reinitialized per round to avoid confirmation bias (Cascante-Bonilla et al., 2020).

b) Committee-Based Iterative Pseudo Labeling

Multiple models (diverse in architecture or initialization) generate pseudo-labels, which are fused using majority voting or confidence aggregation to reduce systematic errors. This robustifies against individual model bias and is especially relevant for structured prediction (e.g., diarization) (Takashima et al., 2021).

c) Iterative Graph or Manifold Methods

Unlabeled data is embedded together with labeled samples; label propagation or optimal transport-based matching is performed in the embedding or structure space; subsequent iterations incorporate only “clean” or mutually consistent assignments, as in iLPC (label cleaning) (Lazarou et al., 2020) or conflict-aware pseudo-labeling via OT (Ding et al., 2022).

d) Teacher-Student and Mean Teacher Frameworks

A “teacher” network (possibly a momentum average of a student) produces pseudo-labels for the student, with cycles of hard label update, soft target interpolation, or exponential averaging over many mini-batches. Such frameworks underpin momentum pseudo-labeling (MPL), slimIPL, and related approaches (Higuchi et al., 2021, Likhomanenko et al., 2020).

e) Iteratively Prompted Foundation Model Supervision

Uses interactions between a trainable “specialist” and a frozen “foundation” model, where the specialist proposes masks or prompts, foundation model generates or refines pseudo-labels, and quality filtering is iteratively tightened (Zhao et al., 1 Apr 2026).

f) Robust Unlabeled Learning for LLM Pseudo-label Denoising

Leverages distribution statistics from two pseudo-labeled sets with different class priors, training a robust classifier to iteratively denoise pseudo-labels via risk estimation, decoupling from LLM confidence (Asano et al., 18 Feb 2025).

3. Mathematical and Algorithmic Formulation

The general iterative pseudo labeling loop can be captured by the following constructs (cf. (Liu et al., 28 May 2026, Xu et al., 2020, Higuchi et al., 2021, Asano et al., 18 Feb 2025, Benato et al., 2021)):

At iteration $t$ , denote labeled set $S$ , unlabeled set $U$ , model parameters $\theta^{(t-1)}$ .
Pseudo-labels are generated as:

$P^{(t)} = \{(x_i, \hat y_i) : x_i \in U, \hat y_i = \arg\max_{y}\; f_{\theta^{(t-1)}}(x_i)_y, \;\text{scoring}\,\geq\tau\}$

possibly after filtering/scoring.

The training goal becomes:

$\min_\theta L(S; \theta) + \lambda_t L(P^{(t)}; \theta) + \mu R(\theta)$

with $L(\cdot)$ denoting supervised or pseudo-labeled task loss (e.g., cross-entropy, CTC loss), $\lambda_t$ an iteration-dependent weight, and $R(\theta)$ a regularizer (e.g., weight decay, KL penalty).

Curriculum/pacing is embodied by adaptation of $\tau$ (confidence/percentile) or $S$ 0 (pseudo-loss weight), e.g., $S$ 1 as label quality improves.
Pseudocode across representative works follows the basic paradigm: $U$ 2 See (Liu et al., 28 May 2026, Takashima et al., 2021, Cascante-Bonilla et al., 2020, Zhao et al., 1 Apr 2026, Lazarou et al., 2020) for detailed, task-specific variants.

4. Domain-Specific Implementations and Empirical Impact

Speech and Language

Iterative pseudo labeling in ASR exploits both traditional CTC/Transducer architectures and instruction-tuned LLM refinement, delivering large word-error-rate (WER) reductions in low-resource and cross-lingual transfer (Xu et al., 2020, Higuchi et al., 2021, Silovsky et al., 2023, Liu et al., 21 Feb 2026). In LLM-centered pipelines, robust unlabeled-unlabeled (UU) learning mitigates overconfidence/confirmation bias, especially for noisy first-pass annotation (Asano et al., 18 Feb 2025).

Vision and Detection

In cross-domain few-shot object detection, iterative pseudo labeling (GiPL) produces substantial mAP gains (>4–6 points over one-shot) by iteratively fusing high-confidence detections with scarce support sets and using curriculum ramping on confidence/weight (Liu et al., 28 May 2026). Seminal work in segmentation (GIST/ RIST) demonstrates that proper alternation schemes—rather than naive ratio mixing—are critical for iterative refinement (Teh et al., 2021).

Structured Data and Knowledge Graph Alignment

Conflict-aware iterative pseudo-labeling via optimal transport in entity alignment ensures one-to-one matching, resolves assignment conflicts, and incrementally refines both candidate matches and global representations, yielding major gains in hit@1 (∼8–10% over prior SOTA) (Ding et al., 2022).

Biomedical and Medical Imaging

Iterative pseudo-label transition mechanisms in tumor segmentation and partial-label medical imaging leverage both teacher-student interpolation and foundation model prompting to recover missing supervision, achieving state-of-the-art Dice and downstream clinical metrics (Jin et al., 6 Aug 2025, Zhao et al., 1 Apr 2026).

5. Hyperparameterization, Convergence, and Practical Recommendations

Hyperparameter choices impact both empirical success and practical convergence:

Number of iterations ( $S$ 2): Most gains accrue in the first 2–5 rounds; improvement plateaus or noise accumulates thereafter (Liu et al., 28 May 2026, Liu et al., 21 Feb 2026, Asano et al., 18 Feb 2025).
Confidence threshold ( $S$ 3): Increasing $S$ 4 improves early reliability, apportioning more data to later rounds as accuracy rises (curriculum). Adaptive or percentile-based thresholds avoid overfitting to initial noise.
Pseudo-loss weight ( $S$ 5): Start with $S$ 6 to suppress noisy pseudo-label impact, anneal to $S$ 7 for stable convergence.
Reinitialization vs. Fine-tuning: Model reinitialization at each round (where feasible) prevents confirmation bias, while continual fine-tuning is standard in domains where catastrophic forgetting is a concern (Cascante-Bonilla et al., 2020, Lazarou et al., 2020, Benato et al., 2021).

Empirical stopping criteria include validation metric delta ( $S$ 8 improvement), maximum iterations, or when newly accepted pseudo-labels fall below a threshold (e.g., $S$ 9) (Benato et al., 2021, Asano et al., 18 Feb 2025).

6. Limitations, Theoretical Considerations, and Future Directions

Despite broad effectiveness, iterative pseudo labeling is subject to several limitations:

Confirmation Bias and Error Accumulation: Poor initial models propagate systematic inaccuracies unless rigorously filtered/weighted, especially under aggressive early $U$ 0, low $U$ 1 (Liu et al., 28 May 2026, Liu et al., 21 Feb 2026).
Class Imbalance and Mode Collapse: Class-balanced normalization (e.g., Sinkhorn-Knopp in iLPC) or row/column matching constraints (in OT-based entity alignment) are required to mitigate degeneracies (Lazarou et al., 2020, Ding et al., 2022).
Overfitting to Pseudo-Labels: Excessive reliance on noisy labels or unrestricted rounds can cause drift or collapse; regularization and quality control (precision-oriented class-wise filtering, entropy checks) are critical (Liu et al., 28 May 2026, Zhao et al., 1 Apr 2026).
Computational Cost: Some variants—particularly those leveraging foundation models or audio-LLM correction—incur nontrivial inference cost (Liu et al., 21 Feb 2026, Zhao et al., 1 Apr 2026).
Rare/Unseen Classes: Extremely infrequent categories may never reach high-confidence pseudo-labels; fusion with true support anchors mitigates but does not eliminate the risk (Liu et al., 28 May 2026, Zhao et al., 1 Apr 2026).

Directions for future research include uncertainty quantification in pseudo-label generation, differentiable end-to-end refinement loops, adaptive committee weighting, multimodal pseudo labeling, and application to diverse, non-English, or highly imbalanced domains (Liu et al., 21 Feb 2026, Asano et al., 18 Feb 2025, Zhao et al., 1 Apr 2026).

References