Class-Mean-Anchored Unlabeled Distillation (CUD)

• The paper introduces CUD, which anchors unlabeled feature representations to stable class means to mitigate catastrophic forgetting in SSCL.
• It leverages cosine similarity and a KL-divergence loss to align current and previous model outputs for reliable unsupervised distillation.
• Integrated within the USP framework, CUD consistently enhances benchmark performance with minimal extra storage and robust hyperparameter stability.

Class-mean-anchored Unlabeled Distillation (CUD) is a distillation technique for semi-supervised continual learning (SSCL) that combats catastrophic forgetting by anchoring the representations of unlabeled data to stable class means derived from labeled examples. CUD is a core module within the USP framework, designed to enhance memory stability without incurring extra labeling or significant memory overhead. Its central mechanism leverages class-means as reliable feature anchors and aligns the similarity distributions of unlabeled data across sequential tasks using a KL-divergence objective, thus extending knowledge distillation to unlabeled domains (Duan et al., 7 Aug 2025).

1. Motivation and Role in Memory Stability

SSCL frameworks confront the dual challenge of learning from sequentially arriving tasks and exploiting both labeled and abundant unlabeled data. While most prior approaches focus distillation strategies exclusively on labeled exemplars, the large pool of unlabeled data remains unregulated; as the model evolves, its feature representations for these unlabeled points can drift, exacerbating forgetting. CUD addresses this deficiency by “anchoring” each unlabeled example to a consistent set of class means—vectors computed only from labeled data, and therefore, more resistant to error accumulation across tasks. The similarity profile of each unlabeled feature to these class means (computed via cosine similarity) is used as the distilled target. By aligning these similarity distributions from the frozen previous model to the current one, memory stability is maintained for unlabeled data, thereby curtailing catastrophic forgetting while not requiring any extra supervision or replay [(Duan et al., 7 Aug 2025), Sec. 3.4].

2. Mathematical Formalism

The mechanism underpinning CUD proceeds as follows:

1. Class Mean Computation For each class $c$ observed up to task $t$, a set of labeled examples $C^{t,(c)}$ is maintained. Features are extracted via a projection head and L₂-normalized:

$f_x^t = P^t(F^t(x)),$

and the class mean is computed as:

$$\mu_{C^{t,(c)}} = \frac{1}{|C^{t,(c)}|} \sum_{x \in C^{t,(c)}} f_x^t.$$

The means are stacked into $M^t \in \mathbb{R}^{d \times k}$.

1. CUD Loss for Unlabeled Data For each unlabeled sample $x_u^t \in D_u^t$:
   • Evaluate similarities, normalized by temperature $\xi$ (default $0.1$), to $M^t$ for both the current (“student”) and the frozen previous (“teacher”) model:

   $t$0

   where $t$1 denotes row-wise cosine similarity. - The CUD objective is applied as:

   $t$2

• The term is weighted by $t$3 (default $t$4), and standard feature normalization is applied throughout [(Duan et al., 7 Aug 2025), Eq. 9, Sec. 3.4].

3. Training Procedure and Pseudocode Integration

CUD operates in tandem with Feature Space Reservation (FSR) and Divide-and-Conquer Pseudo-labeling (DCP) within the broader USP SSCL pipeline. The following outlines the integration:

1. Initialize model with previous parameters, build ETF class prototypes, and compute class means $t$5 from current labeled data.

2. Within each training epoch and batch:

   • Forward features for labeled ($t$6), unlabeled ($t$7), and exemplar data ($t$8)
   • Compute supervised cross-entropy ($t$9), standard CL-distillation ($C^{t,(c)}$0 on exemplars), FSR loss ($C^{t,(c)}$1), and unsupervised loss using DCP ($C^{t,(c)}$2)
   • Compute CUD loss ($C^{t,(c)}$3) by aligning similarity distributions for $C^{t,(c)}$4 with previous features $C^{t,(c)}$5 via KL-divergence, as described above
   • Update total loss as the weighted sum and backpropagate
3. Recompute $C^{t,(c)}$6 each epoch [(Duan et al., 7 Aug 2025), Sec. 3.4 and Appendix A.1].

A high-level pseudocode excerpt:

$$\mu_{C^{t,(c)}} = \frac{1}{|C^{t,(c)}|} \sum_{x \in C^{t,(c)}} f_x^t.$$5

Best practices include detaching previous features/logits, always re-normalizing features, and recomputing $C^{t,(c)}$7 periodically [(Duan et al., 7 Aug 2025), Sec. 6].

4. Theoretical Insights and Intuition

CUD leverages the property that class means, being inferred from labeled exemplars, exhibit substantial resistance to model drift and label noise compared to the neural classifier’s weight vectors under continual learning. By compelling current model features for unlabeled data to preserve their similarity relations to these stable class-means—mirroring the previous model’s relational structure—the framework effectively distills “relational knowledge” about the data manifold. This mechanism is supported by known robustness properties of prototype-based approaches (such as nearest class mean classification) in continual settings. While formal proofs are not provided, the hypothesis is that this prototype-anchored, relational distillation constrains feature space drift in a way that ordinary teacher-student logit matching does not [(Duan et al., 7 Aug 2025), Sec. 3.4].

5. Empirical Validation and Quantitative Effects

Ablation and comparative analyses on standard SSCL benchmarks quantify the contributions of CUD:

• On CIFAR-10 (30 labels/class, 5 tasks), removing CUD (“wo. L_cud”) decrements average accuracy by approximately $C^{t,(c)}$8 percentage points (from $C^{t,(c)}$9 to $f_x^t = P^t(F^t(x)),$0), while DCP alone yields the largest single gain and CUD confers an additional $f_x^t = P^t(F^t(x)),$1–$f_x^t = P^t(F^t(x)),$2 point improvement [(Duan et al., 7 Aug 2025), Fig. 4(a)].
• On CIFAR-100 (25 labels/class, 10 tasks), replacing CUD with ordinary naive logit distillation leads to significantly worse results (average $f_x^t = P^t(F^t(x)),$3 and last accuracy $f_x^t = P^t(F^t(x)),$4) versus CUD’s $f_x^t = P^t(F^t(x)),$5/$f_x^t = P^t(F^t(x)),$6 [(Duan et al., 7 Aug 2025), Sec. 4.2].
• Sensitivity evaluation shows that varying $f_x^t = P^t(F^t(x)),$7 in $f_x^t = P^t(F^t(x)),$8 produces less than $f_x^t = P^t(F^t(x)),$9 percentage points variation around the default, attesting to CUD’s hyperparameter robustness [(Duan et al., 7 Aug 2025), Fig. 8].
• Across all benchmarks, CUD generates gains in average and “last” accuracy within $$\mu_{C^{t,(c)}} = \frac{1}{|C^{t,(c)}|} \sum_{x \in C^{t,(c)}} f_x^t.$$0–$$\mu_{C^{t,(c)}} = \frac{1}{|C^{t,(c)}|} \sum_{x \in C^{t,(c)}} f_x^t.$$1 percentage points, reflecting consistent anti-forgetting benefit (Duan et al., 7 Aug 2025).

6. Practical Implementation Considerations

CUD is implemented in PyTorch as an additional loss component added to the total SSCL objective. All previous model computations are wrapped in torch.no_grad(), class mean matrices are recomputed at least once per epoch, and feature vectors are always L₂-normalized post-operation. No extra storage for prototypes beyond $$\mu_{C^{t,(c)}} = \frac{1}{|C^{t,(c)}|} \sum_{x \in C^{t,(c)}} f_x^t.$$2 is required, and exemplar buffers remain as in standard continual learning pipelines. The reference implementation is publicly available for inspection and experimentation (Duan et al., 7 Aug 2025).

Component	Purpose	Notes
Class-mean matrix $$\mu_{C^{t,(c)}} = \frac{1}{|C^{t,(c)}|} \sum_{x \in C^{t,(c)}} f_x^t.$$3	Stable anchoring for similarity calculation	Recomputed per epoch
CUD loss term	Distill similarity distributions on unlabeled	Weighted by $$\mu_{C^{t,(c)}} = \frac{1}{|C^{t,(c)}|} \sum_{x \in C^{t,(c)}} f_x^t.$$4
Feature normalization	Stability of cosine similarity	Enforced after each projection

The reliance on class-means as the anchor makes this approach lightweight and amenable to integration into existing SSCL architectures.

7. Significance and Context within SSCL

CUD represents an evolution in knowledge distillation for continual learning, extending distillation strategies beyond labeled samples to the dominant, otherwise unregulated, pool of unlabeled data. By leveraging stable class-means, CUD strengthens memory stability without introducing large overheads or necessitating additional storage for pseudo-prototypes. This results in improved balancing of stability and plasticity in realistic semi-supervised continual learning environments. Its integration into the USP framework demonstrably boosts performance over prior art, substantiating the value of prototype-anchored relational distillation strategies (Duan et al., 7 Aug 2025).