Pseudo-Label-Guided Fine-Tuning

• Pseudo-label-guided fine-tuning is a family of methods that reuse model-generated labels to improve prediction accuracy when manual annotations are scarce.
• These techniques employ strategies like confidence thresholding, semantic weighting, geometric priors, and teacher-student frameworks to filter and refine noisy labels.
• Empirical studies across vision, speech, and domain adaptation tasks demonstrate significant performance gains, especially under low-label or domain shift conditions.

Pseudo-label-guided fine-tuning is a family of adaptation procedures in which automatically inferred supervision is reused to optimize a target model when manual annotation is scarce. Across recent work, the inferred target may be a hard class, a soft class distribution, a semantically mixed label, a repaired cross-modal pair, a pruned label set, or a post hoc calibration signal; the unifying principle is that model-generated labels on unlabeled, partially labeled, or weakly labeled data are fed back into training or recalibration to improve the final predictor (Tian et al., 2024, Jin et al., 2022, Chong et al., 2024).

1. Definition and scope

In the literature, pseudo-label-guided fine-tuning appears in several distinct supervision regimes. In semi-supervised learning, a model trained on a small labeled set generates labels for unlabeled examples and is then updated with both sources of supervision. In partial-label and weak-label settings, pseudo-labels disambiguate missing or ambiguous entries and can be followed by category-wise recalibration or iterative self-training. In domain adaptation, pseudo-labels may be enhanced, corrected, or pruned before they are used to adapt a target-domain model. In some cases, the pseudo-label step is itself the main novelty, whereas the downstream optimization remains a standard task loss (Tian et al., 2024, Jin et al., 2022, Xu et al., 17 Oct 2025, Griffin et al., 29 Jun 2026).

Regime	Typical pseudo-label role	Representative instantiations
Semi-supervised classification or segmentation	Direct supervision on unlabeled or mixed samples	Fine-grained SSL, medical segmentation, automatic pixel labeling
Weak or partial supervision	Latent label completion or post hoc recalibration	Partial label learning, multi-label partial labels, 3D weak supervision
Domain adaptation and transfer	Label enhancement, correction, or pruning before downstream training	ASR under accent shift, multi-modal 3D adaptation, zero-shot VLM pruning

This scope is broader than classical self-training. Some methods update the whole model online from current pseudo-labels, as in fine-grained SSL and ASR self-training. Others freeze most of the network and only recalibrate the classifier head, as in Category-wise Fine-Tuning, or edit a pseudo-trained model in parameter space, as in Pseudo2Real (Chong et al., 2024, Lin et al., 9 Oct 2025). A practical implication is that “fine-tuning” in this area does not denote one fixed optimization pattern; it denotes a family of pseudo-supervision-driven adaptation mechanisms.

2. Pseudo-label generation, filtering, and refinement

The most common entry point is confidence-based pseudo-label generation. In PEPL, the model predicts class distributions on unlabeled images, then accepts pseudo-labels only when the maximum confidence exceeds an adaptive per-class threshold built from an EMA global threshold and EMA class-wise confidence statistics. In Filter and Evolve for ASR, each utterance receives a score computed from the average framewise top posterior,

$score = \frac{1}{T}\sum_{i=1}^{T} \max_n C_{n,i},$

and low-score utterances are discarded before CTC fine-tuning (Tian et al., 2024, Jin et al., 2022). In massively multilingual ASR, the crucial design choice is earlier than filtering: the multilingual seed model is first adapted to each target language, and only then used to pseudo-label that language’s unlabeled speech, because direct multilingual pseudo-labeling is poor for low-resource languages (Lugosch et al., 2021).

A second line of work augments confidence with semantic structure. PEPL uses CAMs not to decide the initial class identity, but to estimate how much class-discriminative content survives image mixing; the mixed pseudo-label is weighted by semantic proportions $\rho_a,\rho_b$ rather than by raw area ratio. SemCo adds a second label view in which visually similar classes are grouped, so pseudo-label trust can be evaluated at the group level rather than only at the exact class level. Self-Tuning goes further by avoiding hard pseudo-label cross-entropy on unlabeled data and instead using pseudo-labels to define positive groups in a contrastive objective (Tian et al., 2024, Nassar et al., 2021, Wang et al., 2021).

A third line exploits geometric or contextual priors. SAM-guided pseudo-label enhancement for multi-modal 3D segmentation first assigns a class to each SAM mask by majority vote under size, purity, and representativity constraints, then uses Geometry-Aware Progressive Propagation to spread that label only through geometrically connected 3D points. In 3D weakly supervised segmentation, Class-Aware Label Refinement keeps the top-$V\%$ most confident points per class, after which Geometry-Aware Label Refinement enforces superpoint-level plausibility through a dominant-class ratio $r = \max(A)/\sum(A)$. In noisy correspondence learning, PCSR uses the Pseudo-label Consistency Score

$\mathrm{PCS}_i = n_{\max} - n_{\mathrm{second}}$

to separate noisy pairs into refinable and ambiguous subsets according to temporal stability of pseudo-class predictions across epochs (Yang et al., 2 Feb 2025, Xu et al., 17 Oct 2025, Liu et al., 19 Sep 2025).

Recent work also replaces thresholding with learned judges. TTN formalizes pruning as a Label Imitation Game and trains a task-agnostic “judge” to accept or reject candidate pseudo-labels relative to a reference context, rather than by isolated confidence thresholds. Verifier-guided point tracking learns a separate meta-model that scores candidate trajectories from multiple trackers per frame and selects the most reliable one as pseudo-supervision for real-world adaptation (Griffin et al., 29 Jun 2026, Aydemir et al., 12 Mar 2026). These methods shift pseudo-label selection from scalar confidence to relational inference.

3. How pseudo-labels enter optimization

The simplest pattern is unchanged-task optimization: pseudo-labels are inserted into the same loss used for ground truth. Filter and Evolve uses standard CTC fine-tuning on the union of labeled utterances and filtered pseudo-labeled utterances, with no separate loss for pseudo-labeled data and no consistency term. Automatic pixel-level image labeling reuses standard segmentation cross-entropy, but replaces manual masks with iteratively refined pseudo-masks generated by the previous network state (Jin et al., 2022, Zhang et al., 2020). PEPL also remains close to this template, except that mixed unlabeled data are supervised by a semantic interpolation loss weighted by $\rho_a$ and $\rho_b$, and the total objective is

$$\mathcal{L}_{total} = \gamma \mathcal{L}_{sup} + \lambda \mathcal{L}_{unsup},$$

with $\gamma=\lambda=1$ in the reported experiments (Tian et al., 2024).

Other methods reduce dependence on any one hard pseudo-label. Self-Tuning combines supervised cross-entropy on labeled data with labeled and unlabeled Pseudo Group Contrast; pseudo-labels do not directly impose unlabeled CE targets, but define positive and negative groups in representation space. SURE treats pseudo-label confidences as optimization variables and sharpens them via a maximum infinity-norm regularizer, rather than via a manually chosen confidence threshold. Its unified objective jointly optimizes the model and the latent pseudo-label distribution under simplex and candidate-set constraints (Wang et al., 2021, Feng et al., 2019). This makes pseudo-labeling part of the optimization problem, not a preprocessing heuristic.

Teacher-student organization varies widely. PLGDF uses a mean-teacher V-Net, forms pseudo-labels by averaging teacher predictions under two random-noise perturbations, and supervises both the original unlabeled image and a fused labeled-unlabeled image with the same pseudo-label; sharpening and multi-scale decoder consistency are added through

$$L_{total}=L_{sup}+L_{semi}+\lambda(L_{sharp}+L_{consis}).$$

By contrast, PEPL has no EMA teacher network, and its EMA is used only for adaptive confidence statistics. Massively multilingual ASR uses iterative language-specific self-training before a final multilingual training stage, while Pseudo2Real first fine-tunes source-domain real and source-domain pseudo models from the same initialization, computes

$\rho_a,\rho_b$0

and then corrects the target pseudo-trained model by

$\rho_a,\rho_b$1

These are all pseudo-label-guided fine-tuning schemes, but they intervene at different points of the training stack (Wang et al., 2023, Lugosch et al., 2021, Lin et al., 9 Oct 2025).

4. Architectural patterns and domain-specific variants

A recurring pattern is that architectural novelty is often secondary to pseudo-label construction. PEPL uses an ImageNet-pretrained ResNet-50 and derives CAMs from standard convolutional features. Filter and Evolve builds on a self-supervised pretrained Wav2Vec 2.0 base model with a standard CTC objective. PLGDF uses V-Net with multi-scale decoder outputs, while the Mozambique field-delineation study fine-tunes a FracTAL ResUNet within the DECODE framework. In 3D weak supervision, pseudo-labels generated from OpenSeg or LSeg are fed into a MinkowskiNet18A UNet after class-aware and geometry-aware refinement (Tian et al., 2024, Jin et al., 2022, Wang et al., 2023, Rufin et al., 2023, Xu et al., 17 Oct 2025).

Some variants are explicitly post hoc. Category-wise Fine-Tuning freezes the backbone and recalibrates the classification layer only, viewing the head as $\rho_a,\rho_b$2 independent logistic regressions. Calibration is performed either with BP-ASL on known labels for each category or with a Genetic Algorithm that directly maximizes AUC or AP on the known-label subset. The point is not to regenerate better pseudo-labels, but to repair the final classifier after pseudo-label-based training has already injected noise (Chong et al., 2024). A related but parameter-space-oriented view appears in Pseudo2Real, where the pseudo-to-real correction vector is learned in a labeled source domain and then added to a target-domain pseudo-trained ASR model (Lin et al., 9 Oct 2025).

Other methods operate even earlier, before the downstream learner is retrained. SAM-guided pseudo-label enhancement plugs into xMUDA or SUMMIT and densifies sparse target pseudo-labels before adaptation. TTN prunes hallucinated pseudo-labels from VLM outputs before a downstream detector is trained. Verifier-guided point tracking does the same for trajectory candidates: it does not alter the downstream Track-On2 architecture, but improves the quality of real-world pseudo-labels used to fine-tune it (Yang et al., 2 Feb 2025, Griffin et al., 29 Jun 2026, Aydemir et al., 12 Mar 2026). This suggests a useful architectural taxonomy: pseudo-label-guided fine-tuning may target the label generator, the label filter, the training loss, the classifier head, or the full parameter vector.

5. Empirical behavior across tasks

The strongest gains are usually reported in low-label or strong-shift regimes. On CUB_200_2011, PEPL reports accuracies of 38.53, 64.60, and 76.97 for 10%, 20%, and 30% label ratios, compared with FreeMatch’s 30.78, 56.68, and 67.62 and supervised-only’s 28.61, 51.87, and 65.77; on Stanford Cars, it reports 32.72, 74.79, and 86.52 versus FreeMatch’s 26.10, 62.67, and 75.97. Its semantic-aware ablation on CUB reaches 38.53, 64.60, 76.97, and 88.75, versus 27.47, 56.23, 72.50, and 85.08 without semantic awareness (Tian et al., 2024). These results are consistent with the paper’s claim that fine-grained pseudo-labeling benefits when semantic localization, rather than only confidence, is modeled.

ASR work shows a similar pattern. In Filter and Evolve, LS-100/LS-860 improves from 6.10/13.30 WER for supervised fine-tuning to 5.10/10.84 with score filtering, while LS-10/LS-90 improves from 11.10/17.60 to 7.67/13.86. In massively multilingual ASR, the final multilingual model trained from scratch on Common Voice plus pseudo-labeled VoxPopuli and then fine-tuned on Common Voice reaches 10.6 test CER averaged over the 19 CV∩VP languages, versus 24.8 for the base multilingual model. Pseudo2Real reports the largest relative effect: on AfriSpeech-200 with Whisper tiny, average WER drops from 89.3 for target pseudo-label fine-tuning to 57.7 after parameter-space correction, a relative reduction of about 35.4% (Jin et al., 2022, Lugosch et al., 2021, Lin et al., 9 Oct 2025).

Outside conventional SSL, pseudo-label-guided fine-tuning also yields strong post hoc and source-free gains. Category-wise Fine-Tuning reports 91.82% single-model mAUC on CheXpert, 83.69% average mAP on partially labeled MS-COCO, and 85.31% mAP on Open Images V3. In source-free field delineation, pseudo-label-only fine-tuning recovers up to 77% of the mIoU increase and 68% of the mRMSE decrease obtained by human labels, and adding pseudo labels to human labels further raises Mozambique performance from $\rho_a,\rho_b$3 to $\rho_a,\rho_b$4 and improves $\rho_a,\rho_b$5 from 0.040 to 0.037 ha (Chong et al., 2024, Rufin et al., 2023).

Verification-style methods show a different empirical signature: modest average gains but disproportionately large improvements on hard cases. TTN raises overall pseudo-label $\rho_a,\rho_b$6 from 0.362 for the auto-labeling baseline to 0.376, and TTN$\rho_a,\rho_b$7 to 0.387; for the worst-performing baseline categories, mean $\rho_a,\rho_b$8 rises by 28% with zero-shot TTN and by 44% with TTN$\rho_a,\rho_b$9, and the paper reports Category Revival on LVIS, where training on pruned labels recovers categories that otherwise had zero recall (Griffin et al., 29 Jun 2026). In real-world point tracking, verifier-guided pseudo-labeling improves Track-On2 on all four real benchmarks, including an increase from 61.7 to 67.3 average on EgoPoints and from 68.1 to 70.9 AJ on RoboTAP (Aydemir et al., 12 Mar 2026). These results suggest that pseudo-label quality control is particularly valuable when raw pseudo-labels are already numerous but structurally noisy.

6. Boundaries, misconceptions, and open issues

A common misconception is that pseudo-label-guided fine-tuning is synonymous with confidence-thresholded self-training. The surveyed methods contradict that view. Some use thresholding, but others use CAM-derived semantic weights, grouped label views, superpoint geometry, temporal stability scores, adversarial-style judges, or verifier-selected multi-teacher trajectories. Some methods are post hoc rather than end-to-end, such as Category-wise Fine-Tuning. Others are adjacent rather than literal fine-tuning: the rubric-guided video summarization framework uses pseudo labels to induce prompts and rubrics for zero-shot inference, but explicitly does not perform parameter updates (Chong et al., 2024, Wu et al., 20 Oct 2025, Griffin et al., 29 Jun 2026).

Another misconception is that better pseudo-labels always require stronger architectures. Several papers argue the opposite: the main leverage often comes from supervision design. PEPL attributes its gains to semantic-aware mixed-label construction rather than architectural complexity; SAM-guided enhancement keeps xMUDA and SUMMIT unchanged and only replaces their pseudo-labels; verifier-guided point tracking leaves Track-On2 intact and alters only the label generator (Tian et al., 2024, Yang et al., 2 Feb 2025, Aydemir et al., 12 Mar 2026). This suggests that in many regimes, pseudo-label-guided fine-tuning is bottlenecked less by backbone capacity than by whether noisy auto-labels are admitted, reweighted, repaired, or discarded correctly.

The main failure modes are equally recurrent. Early pseudo-label errors can propagate through later rounds of self-training; CAMs or SAM masks may be poorly localized; global thresholds can create class imbalance; categories with very few known labels can make category-wise recalibration unstable; correction vectors may not transfer if pseudo-label bias is not stationary across domains; and verifier-style selection is bounded by the quality of the candidate teacher pool (Tian et al., 2024, Yang et al., 2 Feb 2025, Lin et al., 9 Oct 2025, Xu et al., 17 Oct 2025, Griffin et al., 29 Jun 2026, Aydemir et al., 12 Mar 2026). Open problems stated or implied across the literature include more principled confidence estimators than simple average posterior scores, better handling of categories with extremely sparse clean labels, stronger theory for when source-domain pseudo-to-real corrections transfer, and more systematic study of how pseudo-label pruning, geometry-aware propagation, and iterative retraining should be combined (Jin et al., 2022, Chong et al., 2024, Lin et al., 9 Oct 2025, Rufin et al., 2023).

Taken together, the literature defines pseudo-label-guided fine-tuning not as one algorithm but as a design space. The central technical question is not whether pseudo-labels should be used, but how their reliability should be estimated, how their errors should be constrained, and at which point in the adaptation pipeline they should influence optimization.