Divide-and-Conquer Pseudo-labeling (DCP)

• The paper introduces DCP with a novel divide-and-conquer strategy that combines classifier-driven high-confidence pseudo-labeling with NCM-based labeling for low-confidence samples.
• DCP partitions unlabeled data using a fixed confidence threshold, enabling more effective utilization of abundant unlabeled examples while reducing noise.
• Integration of DCP within the USP framework supports both learning plasticity and memory stability, leading to notable performance improvements in SSCL.

Divide-and-Conquer Pseudo-labeling (DCP) is a technique introduced within the USP framework for semi-supervised continual learning (SSCL). DCP addresses the challenge of leveraging unlabeled data in sequential learning tasks, specifically focusing on improving pseudo-labeling reliability across both high- and low-confidence predictions. It combines outputs from the standard classifier with Nearest-Class-Mean (NCM) assignment to more effectively utilize unlabeled data while supporting both learning plasticity (accommodating new classes) and memory stability (retaining prior knowledge) (Duan et al., 7 Aug 2025).

1. Problem Context and Motivation

Semi-supervised continual learning (SSCL) is characterized by the sequential arrival of both labeled and unlabeled data over multiple tasks. At each task $t$, the available data consists of a small labeled set $D^t_l=\{(x_i, y_i)\}$ and a generally much larger unlabeled set $D^t_u=\{x_i\}$. The dual goals are plasticity—efficiently learning new concepts—and stability—avoiding catastrophic forgetting of previous knowledge. Pseudo-labeling methods attempt to exploit $D^t_u$ by assigning artificial labels, typically via model self-prediction (softmax probabilities). However, these methods face a key limitation: standard thresholded classifier pseudo-labels are reliable only when the model is confident, thereby discarding a substantial proportion of the unlabeled data. Naively including low-confidence samples introduces noise and degrades learning. DCP directly targets this challenge by partitioning unlabeled data based on model confidence and applying a divide-and-conquer pseudo-labeling strategy.

2. DCP Pipeline and Mechanism

For each minibatch of unlabeled data $x \in D^t_u$, DCP proceeds as follows:

• Compute the normalized feature vector:

$$f_x = \frac{P(F(x))}{\|P(F(x))\|_2} \in \mathbb{R}^d$$

Here, $F$ is the feature extractor and $P$ is a projection head.

• Compute the softmax class probabilities:

$$p(x) = [p_1, ..., p_{K_{\le t}}], \quad p_k = p(y=k \mid x)$$

• Calculate the confidence score:

$c(x) = \max_{1 \le k \le K_{\le t}} p_k(x)$

• Divide $D^t_l=\{(x_i, y_i)\}$0 into two disjoint subsets:

$D^t_l=\{(x_i, y_i)\}$1

where $D^t_l=\{(x_i, y_i)\}$2 is a fixed confidence threshold.

• For high-confidence samples $D^t_l=\{(x_i, y_i)\}$3, use the classifier’s hard pseudo-label:

$D^t_l=\{(x_i, y_i)\}$4

• For low-confidence samples $D^t_l=\{(x_i, y_i)\}$5, assign the label by a Nearest-Class-Mean (NCM) classifier:

$D^t_l=\{(x_i, y_i)\}$6

where $D^t_l=\{(x_i, y_i)\}$7 is the cosine similarity, and $D^t_l=\{(x_i, y_i)\}$8 is the mean feature vector of the labeled exemplars $D^t_l=\{(x_i, y_i)\}$9 for class $D^t_u=\{x_i\}$0:

$D^t_u=\{x_i\}$1

3. Mathematical Loss Formulation and Optimization

The standard FixMatch loss for unsupervised learning considers only high-confidence pseudo-labels:

$D^t_u=\{x_i\}$2

where $D^t_u=\{x_i\}$3 denotes strong augmentation, and $D^t_u=\{x_i\}$4 is cross-entropy. DCP generalizes this to:

$D^t_u=\{x_i\}$5

This formulation exploits high-confidence pseudo-labels when available but repurposes low-confidence samples via exemplar-driven assignment rather than discarding them.

During back-propagation, $D^t_u=\{x_i\}$6 is averaged over the batch and summed with other supervised and distillation losses that comprise the full USP objective.

4. Pseudocode for Implementation

The stepwise execution of DCP for each unlabeled minibatch can be summarized as:

$D^t_u$1

5. Integration with Broader USP Framework

DCP forms one component of the three-pronged USP architecture, targeting enhancement of unlabeled learning (UL). Outputs from DCP are also utilized downstream by the CUD (Class-mean-anchored Unlabeled Distillation) subcomponent. Specifically, the feature embeddings $D^t_u=\{x_i\}$7 as well as the current class-mean matrix $D^t_u=\{x_i\}$8—maintained from labeled exemplars—are passed to CUD. CUD then enforces consistency between the current model’s feature-to-mean similarities and those of a frozen previous model, anchoring unlabeled data to stable class means via a KL divergence loss:

$D^t_u=\{x_i\}$9

This dual use of DCP outputs supports both robust pseudo-label quality and longitudinal retention of learned representations (Duan et al., 7 Aug 2025).

6. Empirical Assessment and Comparative Performance

Ablation studies indicate substantial performance benefits for SSCL when employing DCP. For instance, removing DCP ("wo. $D^t_u$0") leads to a decrease in final 5-task CIFAR10-30 accuracy from approximately 81.4% to 68.3%. Substituting DCP with either classifier-only or NCM-only pseudo-labeling degrades performance further. Moreover, across tasks, DCP achieves pseudo-label accuracy on low-confidence samples that is 10–20% higher than the standard classifier’s outputs. This improvement on the challenging subset explains its contribution to overall accuracy gains (Duan et al., 7 Aug 2025).

7. Significance and Theoretical Implications

By partitioning the unlabeled set according to prediction confidence and adapting label assignment strategies accordingly, DCP enables models to utilize a broader spectrum of the unlabeled data without introducing excessive noise. The divide-and-conquer mechanism—classifier-driven for high-confidence and exemplar-driven for low—exemplifies a targeted approach to dealing with the inherent uncertainty in pseudo-labeling. This construct not only augments effective learning in the current task but also provides stable anchors for continual distillation, supporting both plasticity and stability. A plausible implication is that similar mechanisms could generalize to other semi-supervised or continual learning contexts where labeled data is scarce and confidence distributions are heavy-tailed.