Self-Improvement via Unlabeled Data

Updated 16 November 2025

Self-Improvement via Unlabeled Data is a method that uses iterative pseudo-labeling and self-supervision to enhance model performance without manual annotations.
It employs techniques such as learned thresholding and consistency regularization to improve accuracy, generalization, and robustness across diverse modalities.
Empirical studies demonstrate significant error reductions and scalability benefits, making it a practical approach for real-world applications in vision, language, and beyond.

The term "self-improvement via unlabeled data" refers to algorithmic strategies through which a model iteratively increases its accuracy, generalization, or robustness by extracting learning signals from data without human-provided labels. In contrast to classical supervised learning—where improvement is strictly tied to curated annotations—self-improvement methods leverage pseudo-labeling, self-supervision, distributional heuristics, and related frameworks to unlock the information content of large, annotation-missing corpora. Empirical and theoretical results across modalities (vision, language, structured data, RL, graphs) demonstrate that, if properly filtered, leveraged, or regularized, unlabeled data can serve not only as a substitute for labels but often as an amplifier of a model’s own capacity to generalize and adapt.

1. Core Methodologies for Harnessing Unlabeled Data

Common self-improvement pipelines with unlabeled data follow an iterative, pseudo-labeling paradigm:

Iterative Self-Training: At each round, a model trained on available labels is used to generate pseudo-labels for high-confidence unlabeled instances, which are then incorporated into the training set. This expands the "clean" training set and allows updating the model over many cycles (Dupre et al., 2019).
Ensemble Pseudo-Labeling and Learned Thresholding: Multiple augmentations of each unlabeled sample are scored, and a confidence metric (often a weighted sum of softmax-based scores, margin, and distributional distance) is used to select candidates; calibration is enforced via thresholds learned to guarantee a target pseudo-labeling precision (Dupre et al., 2019).
Self-Supervision and Consistency Regularization: Models are trained to produce invariant representations under data augmentations or designed pretext tasks (e.g., rotation prediction, jigsaw, transformation discrimination), either in pre-training or coupled with the self-training loop (Sahito et al., 2021, Banitalebi-Dehkordi et al., 2022).
Curriculum and Density-Aware Selection: Pseudo-label selection can be scheduled (percentile-based curriculum) and regularized by penalizing pseudo-labels lying in low-density regions, thereby enforcing the cluster assumption (Kim et al., 2023).
Self-Distillation and Multi-Model Consensus: Teacher-student structures with independent or differently paced learners enforce agreement (often via MSE or KL regularization), encouraging stable pseudo-labeling and discouraging confirmation bias (Chen et al., 2020).
Domain-Agnostic and Streaming Self-Training: Unlabeled data is processed in domain-agnostic streams, with pseudo-label pretraining and subsequent fine-tuning on labeled data, thus enabling continuous self-improvement even in the presence of significant distribution shift (Lin et al., 2021).

2. Mathematical and Algorithmic Foundations

The essence of self-improvement via unlabeled data lies in recasting unlabeled data as a supervisory resource through iterative approximation of the risk minimization objective:

Pseudo-Labeled Empirical Risk: At each iteration, a mixed loss is minimized:

$\mathcal{L}(\theta) = \sum_{(x,y)\in D_l} \ell(y, f_\theta(x)) + \lambda \sum_{x\in D_u'} \ell(\hat{y}(x), f_\theta(x))$

where $D_l$ is the labeled set, $D_u'$ the selected pseudo-labeled set, and $\hat{y}(x)$ the current pseudo-label for $x$ .

Learned Thresholding for Label Addition:

$T_c = \max\{ t : \text{acc}(D_l, \theta, t) \geq T_a \}$

ensures high-precision pseudo-label addition by tuning the confidence threshold to match a target accuracy $T_a$ evaluated on held-out labeled data (Dupre et al., 2019).

Ensemble Metrics: Candidate scores may combine softmax confidence ( $c_a$ ), prediction margin ( $c_b$ ), and distributional distance to class-wise posteriors ( $c_c$ ):

$c_\text{total}(x) = w_a c_a + w_b c_b + w_c c_c$

Theoretical Convergence: In neural network settings, generalization error and convergence rate can be shown to improve as $O(1/\sqrt{M})$ in the number $M$ of unlabeled samples used for self-training (Zhang et al., 2022). Empirically, each addition of $\approx\sqrt{M}$ examples yields the same incremental boost, with diminishing returns as $M$ increases.
Consistency Regularization and Self-Supervision: Auxiliary losses enforce invariance of prediction under data transformation or maximize entropy for out-of-distribution auxiliary samples (Banitalebi-Dehkordi et al., 2022, Nadimpalli et al., 2021).
Density/Affinity Constraints: Regularization of pseudo-label selection scores with (normalized) likelihoods or prototype affinity constrains the model to exploit high-density, manifold-aligned regions, reducing error propagation (Kim et al., 2023, Banitalebi-Dehkordi et al., 2022).

3. Empirical Gains and Scaling Results

Self-improvement via unlabeled data consistently delivers measurable performance gains across varied settings:

Method/Dataset	Baseline Error (%)	With Self-Improvement (%)	Samples Added (%)
CIFAR-100 (ResNet-18) (Dupre et al., 2019)	32.49	28.09	75
TinyImageNet (ResNet-18) (Dupre et al., 2019)	37.47	33.68	81

On CIFAR-100/TinyImageNet, error rates are reduced by 4–5 points and the labeled set is effectively increased by 70–80% via pseudo-label cycles over 20–30 iterations (Dupre et al., 2019).
Cross-lingual transfer: Iterative self-training with pseudo-labeled, high-confidence spans in the target language boosts F1 by up to 8.7 points over zero-shot baselines (Huang et al., 2021).
Tabular data: Consistent improvement of up to 29% F1 in label-scarce regimes through curriculum and density-regularized pseudo-labeling (Kim et al., 2023).
Image classification, graphs, HAR: Integration with self-supervised losses or consistency terms pushes mean accuracy up to 13 points or reduces error rate by 10 points, often matching or exceeding baseline supervised runs using all available labels (Sahito et al., 2021, Tang et al., 2021, Liu et al., 2023).
Adversarial robustness: Robust self-training with pseudo-labeled inputs allows matching the robust accuracy of fully supervised models at a fraction of label cost; e.g., on CIFAR-10, robust accuracy rises 7.7 points over best supervised adversarial training (Carmon et al., 2019).

Scalability trends:

Dataset growth is geometric for early iterations, moderate for late iterations.
Training time per iteration grows linearly in the labeled pool's size; overall computational cost remains modest (≈50% over baseline) if batch size is held constant (Dupre et al., 2019).

4. Failure Modes, Limitations, and Open-World Adaptivity

Systematic limitations and failure modes include:

Class Imbalance: When in-distribution labeled or unlabeled pools are imbalanced, the thresholding and pseudo-label selection may over-fit to majority classes. The method is sensitive to the initial labeled size; <10 samples per class may result in poorly calibrated confidence metrics and label drift (Dupre et al., 2019).
Fine-Grained or Low-Resolution Domains: On fine-grained classification or similar-featured classes, distributional distance metrics cease to discriminate effectively, leading to mislabeling (Dupre et al., 2019).
Noisy or Open-Set Inputs: In presence of substantial out-of-distribution data, naive self-training admits erroneous pseudo-labels due to over-confidence; targeted outlier-aware sample selection and entropy maximization on negatives are essential (Banitalebi-Dehkordi et al., 2022, Augustin et al., 2020).
Overfitting/Confirmation Bias: Excessive epochs before pseudo-label expansion or use of stale model weights may cause the variance across augmentations to collapse, allowing the system to select and reinforce incorrect samples (Radhakrishnan et al., 2023).

Advanced variants mitigate open-world and confirmation bias phenomena by introducing:

Out-distribution-aware thresholds using in- and out-domain validation splits (Augustin et al., 2020).
Adaptive sample filtering using representation distances and calibrated entropy/softmax confidence (Banitalebi-Dehkordi et al., 2022, Radhakrishnan et al., 2023).
Self-distillation and multi-teacher schemes to enforce consensus and decorrelate confirmation bias across self-training rounds (Chen et al., 2020).

5. Theoretical Insights and General Principles

Recent theoretical work has clarified the fundamental mechanisms through which unlabeled data improves self-supervised training:

Local Risk Landscape Smoothing: Incorporation of pseudo-labeled unlabeled samples shifts the empirical risk minimizer from an initialization-biased point towards the true underlying minimizer, both accelerating training and reducing generalization error (Zhang et al., 2022).
Convergence Rate: For ReLU networks, the theoretical contraction rate per self-training iteration and fixed-point error both improve as $O(1/\sqrt{M})$ , with $M$ the number of unlabeled examples. Each additional $O(\sqrt{M})$ samples yields diminishing but positive returns, consistent with empirical findings in large-scale semi-supervised learning (Zhang et al., 2022).
Unlabeled Data as Proxy for Labels: Provided pseudo-labels are suitably high-confidence and class balance is respected, unlabeled examples can serve as effective proxies for labeled samples, narrowing the generalization gap and, under cluster assumptions or mild distribution shift, yielding faster error decay than standard ERM (Saberi et al., 2023).

6. Application to Diverse Learning Modalities

Self-improvement via unlabeled data admits broad adaptation:

Vision: Iterative expansion of labeled data for image classification and adversarial robustness (Dupre et al., 2019, Carmon et al., 2019); open-world robustness to unrelated data via outlier-aware sample selection (Augustin et al., 2020); consistency/self-supervised learning for feature invariance (Banitalebi-Dehkordi et al., 2022, Sahito et al., 2021).
Text and NLP: Task-specific pretraining using pseudo-labeled domain text between initial pretraining and fine-tuning (Guo, 2020); cross-lingual transfer by generating pseudo-answers in unlabeled target language contexts (Huang et al., 2021); curriculum-based or density-regularized pseudo-label selection for tabular/structured NLP (Kim et al., 2023).
Graphs: Diffusion-based self-improvement in property prediction by synthesizing task-aligned, diverse, and label-consistent graph augmentations from a large, unlabeled corpus (Liu et al., 2023).
Reinforcement Learning: Pre-training low-level skills with VAE on unlabeled trajectories and re-labeling prior trajectories provides a strong foundation for hierarchical RL, outperforming baselines in long-horizon, sparse-reward tasks (Wilcoxson et al., 23 Oct 2024).
Other Modalities: Audio-LMs, activity recognition, and large-scale tabular classifiers benefit from tailored self-training plus consistency/self-supervision, often with little or no increase in inference-time complexity (Wang et al., 27 Jul 2025, Tang et al., 2021, Nadimpalli et al., 2021).

7. Practitioner Guidance and Methodological Recommendations

Key practical recommendations for maximizing self-improvement from unlabeled data include:

Model Calibration: Always learn or adapt pseudo-labeling thresholds based on held-out labeled data and explicit calibration (e.g., temperature scaling), enforcing a high-precision regime in early expansion (Dupre et al., 2019, Radhakrishnan et al., 2023).
Augmentation Diversity: Employ ensembles of data augmentations and select samples based on consensus and stability metrics, not simply maximal softmax probability (Dupre et al., 2019, Banitalebi-Dehkordi et al., 2022).
Regularization and Curriculum: Couple consistency losses, curriculum schedules, and cluster- or density-aware scoring to prevent accumulation of spurious pseudo-labels and to maintain class balance (Kim et al., 2023, Banitalebi-Dehkordi et al., 2022).
Validation and Monitoring: Track true- versus pseudo-label accuracy on a small validation set at each iteration to detect drift; halt expansion if new examples are not reliably correct (Dupre et al., 2019, Hyams et al., 2017).
Open-World Robustness: In open-set scenarios, combine representation-based outlier filtering, entropy-maximization regularization, and out-distribution-aware sample selection (Augustin et al., 2020, Banitalebi-Dehkordi et al., 2022, Radhakrishnan et al., 2023).
Scale and Compute: While large unlabeled pools require increased compute, per-iteration cost is typically linear in the training set size, and data expansion yields diminishing but continual returns (Dupre et al., 2019).

References

"Iterative Self-Learning: Semi-Supervised Improvement to Dataset Volumes and Model Accuracy" (Dupre et al., 2019)
"How does unlabeled data improve generalization in self-training? A one-hidden-layer theoretical analysis" (Zhang et al., 2022)
"Enhancing Self-Training Methods" (Radhakrishnan et al., 2023)
"AuxMix: Semi-Supervised Learning with Unconstrained Unlabeled Data" (Banitalebi-Dehkordi et al., 2022)
"Revisiting Self-Training with Regularized Pseudo-Labeling for Tabular Data" (Kim et al., 2023)
"Out-distribution aware Self-training in an Open World Setting" (Augustin et al., 2020)
"Better Self-training for Image Classification through Self-supervision" (Sahito et al., 2021)
"Harnessing Unlabeled Data to Improve Generalization of Biometric Gender and Age Classifiers" (Nadimpalli et al., 2021)
"Unlabeled Data Improves Adversarial Robustness" (Carmon et al., 2019)
"Streaming Self-Training via Domain-Agnostic Unlabeled Images" (Lin et al., 2021)
"Data-Centric Learning from Unlabeled Graphs with Diffusion Model" (Liu et al., 2023)
"SelfHAR: Improving Human Activity Recognition through Self-training with Unlabeled Data" (Tang et al., 2021)
"Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration" (Wilcoxson et al., 23 Oct 2024)
"Self-Improvement for Audio LLM using Unlabeled Speech" (Wang et al., 27 Jul 2025)
"Self-PU: Self Boosted and Calibrated Positive-Unlabeled Training" (Chen et al., 2020)

The cumulative evidence demonstrates that with algorithmic rigor—careful confidence estimation, robust loss design, and iterated calibration—self-improvement via unlabeled data is both effective and broadly applicable across modalities, offering systematic advances in both quantity and quality of learned models.