---
title: Divide-and-Conquer Pseudo-labeling (DCP)
url: https://www.emergentmind.com/topics/divide-and-conquer-pseudo-labeling-dcp
type: topic
---

# Divide-and-Conquer Pseudo-labeling (DCP)

Divide-and-Conquer Pseudo-labeling (DCP) is a technique introduced within the USP framework for semi-supervised continual learning (SSCL). DCP addresses the challenge of leveraging unlabeled data in sequential learning tasks, specifically focusing on improving pseudo-labeling reliability across both high- and low-confidence predictions. It combines outputs from the standard classifier with Nearest-Class-Mean (NCM) assignment to more effectively utilize unlabeled data while supporting both learning plasticity (accommodating new classes) and memory stability (retaining prior knowledge) [2508.05316].

## 1. Problem Context and Motivation

Semi-supervised continual learning (SSCL) is characterized by the sequential arrival of both labeled and unlabeled data over multiple tasks. At each task $t$, the available data consists of a small labeled set $D^t_l=\{(x_i, y_i)\}$ and a generally much larger unlabeled set $D^t_u=\{x_i\}$. The dual goals are plasticity—efficiently learning new concepts—and stability—avoiding catastrophic forgetting of previous knowledge. Pseudo-labeling methods attempt to exploit $D^t_u$ by assigning artificial labels, typically via model self-prediction (softmax probabilities). However, these methods face a key limitation: standard thresholded classifier pseudo-labels are reliable only when the model is confident, thereby discarding a substantial proportion of the unlabeled data. Naively including low-confidence samples introduces noise and degrades learning. DCP directly targets this challenge by partitioning unlabeled data based on model confidence and applying a divide-and-conquer pseudo-labeling strategy.

## 2. DCP Pipeline and Mechanism

For each minibatch of unlabeled data $x \in D^t_u$, DCP proceeds as follows:

- Compute the normalized feature vector:
  $$
  f_x = \frac{P(F(x))}{\|P(F(x))\|_2} \in \mathbb{R}^d
  $$
  Here, $F$ is the feature extractor and $P$ is a projection head.
- Compute the softmax class probabilities:
  $$
  p(x) = [p_1, ..., p_{K_{\le t}}], \quad p_k = p(y=k \mid x)
  $$
- Calculate the confidence score:
  $$
  c(x) = \max_{1 \le k \le K_{\le t}} p_k(x)
  $$
- Divide $D^t_u$ into two disjoint subsets:
  $$
  \mathcal{H} = \{ x \mid c(x) \ge \tau \},\quad
  \mathcal{L} = \{ x \mid c(x) < \tau \}
  $$
  where $\tau$ is a fixed confidence threshold.

- For high-confidence samples $x \in \mathcal{H}$, use the classifier’s hard pseudo-label:
  $$
  \hat{y}(x) = \arg\max_k p_k(x)
  $$
- For low-confidence samples $x \in \mathcal{L}$, assign the label by a Nearest-Class-Mean (NCM) classifier:
  $$
  q(x) = \arg\max_{i \in \{1, \ldots, K_{\le t}\}} S(f_x, \mu_i)
  $$
  where $S(u, v) = u^\top v$ is the cosine similarity, and $\mu_i$ is the mean feature vector of the labeled exemplars $E_i$ for class $i$:
  $$
  \mu_i = \frac{1}{|E_i|}\sum_{x \in E_i} f_x
  $$

## 3. Mathematical Loss Formulation and Optimization

The standard FixMatch loss for unsupervised learning considers only high-confidence pseudo-labels:
$$
\mathcal{L}_{\mathrm{uns}}
= \mathbb{E}_{x\in D^t_u} \big[\, \mathbf{1}(c(x)\ge\tau)\, H(p(\alpha(x)), \hat y(x)) \big]
$$
where $\alpha(\cdot)$ denotes strong augmentation, and $H(p,y) = -\sum_i \mathbf{1}(i = y) \log p_i$ is cross-entropy. DCP generalizes this to:
$$
\mathcal{L}'_{\mathrm{uns}}
= \mathbb{E}_{x\in D^t_u}
\big[
\mathbf{1}(c(x)\ge\tau)\, H(p(\alpha(x)), \hat y(x))
+ \mathbf{1}(c(x)<\tau)\, H(p(\alpha(x)), q(x))
\big]
$$
This formulation exploits high-confidence pseudo-labels when available but repurposes low-confidence samples via exemplar-driven assignment rather than discarding them.

During back-propagation, $\mathcal{L}'_{\mathrm{uns}}$ is averaged over the batch and summed with other supervised and distillation losses that comprise the full USP objective.

## 4. Pseudocode for Implementation

The stepwise execution of DCP for each unlabeled minibatch can be summarized as:

```python
Input:
    minibatch of unlabeled x ∈ D^t_u
    threshold τ
    feature-extractor F, projection P, classifier G
    class-means {μ_i} from labeled exemplars
Output:
    unsupervised loss ℒ'_uns

ℒ'_uns ← 0
for each x in minibatch:
    f ← normalize(P(F(x)))
    p ← softmax(G(F(x)))        # p ∈ R^K
    c ← max(p)
    x̄ ← strong_augment(x)
    p̄ ← softmax(G(F(x̄)))
    if c ≥ τ:                   # high-confidence region 𝓗
        ŷ ← argmax_k p_k
        ℒ'_uns += CrossEntropy(p̄, ŷ)
    else:                       # low-confidence region 𝓛
        q ← argmax_i fᵀ μ_i
        ℒ'_uns += CrossEntropy(p̄, q)
    end if
ℒ'_uns ← ℒ'_uns / batch_size
```

## 5. Integration with Broader USP Framework

DCP forms one component of the three-pronged USP architecture, targeting enhancement of unlabeled learning (UL). Outputs from DCP are also utilized downstream by the CUD (Class-mean-anchored Unlabeled Distillation) subcomponent. Specifically, the feature embeddings $f_x$ as well as the current class-mean matrix $M = [\mu_1, ..., \mu_K]$—maintained from labeled exemplars—are passed to CUD. CUD then enforces consistency between the current model’s feature-to-mean similarities and those of a frozen previous model, anchoring unlabeled data to stable class means via a KL divergence loss:
$$
\mathcal{L}_{\mathrm{cud}}
= \mathbb{E}_{x\in D^t_u}
\mathrm{KL} \Big(
\mathrm{softmax}\big(S(f_x, M) / \xi\big)
\parallel
\mathrm{softmax}\big(S(f^{\,\mathrm{old}}_x, M) / \xi\big)
\Big)
$$
This dual use of DCP outputs supports both robust pseudo-label quality and longitudinal retention of learned representations [2508.05316].

## 6. Empirical Assessment and Comparative Performance

Ablation studies indicate substantial performance benefits for SSCL when employing DCP. For instance, removing DCP ("wo. $\mathcal{L}'_{\mathrm{uns}}$") leads to a decrease in final 5-task CIFAR10-30 accuracy from approximately 81.4% to 68.3%. Substituting DCP with either classifier-only or NCM-only pseudo-labeling degrades performance further. Moreover, across tasks, DCP achieves pseudo-label accuracy on low-confidence samples that is 10–20% higher than the standard classifier’s outputs. This improvement on the challenging subset explains its contribution to overall accuracy gains [2508.05316].

## 7. Significance and Theoretical Implications

By partitioning the unlabeled set according to prediction confidence and adapting label assignment strategies accordingly, DCP enables models to utilize a broader spectrum of the unlabeled data without introducing excessive noise. The divide-and-conquer mechanism—classifier-driven for high-confidence and exemplar-driven for low—exemplifies a targeted approach to dealing with the inherent uncertainty in pseudo-labeling. This construct not only augments effective learning in the current task but also provides stable anchors for continual distillation, supporting both plasticity and stability. A plausible implication is that similar mechanisms could generalize to other semi-supervised or continual learning contexts where labeled data is scarce and confidence distributions are heavy-tailed.

Source: https://www.emergentmind.com/topics/divide-and-conquer-pseudo-labeling-dcp