---
title: Class-Mean-Anchored Unlabeled Distillation (CUD)
url: https://www.emergentmind.com/topics/class-mean-anchored-unlabeled-distillation-cud
type: topic
---

# Class-Mean-Anchored Unlabeled Distillation (CUD)

Class-mean-anchored Unlabeled Distillation (CUD) is a distillation technique for semi-supervised continual learning (SSCL) that combats catastrophic forgetting by anchoring the representations of unlabeled data to stable class means derived from labeled examples. CUD is a core module within the USP framework, designed to enhance memory stability without incurring extra labeling or significant memory overhead. Its central mechanism leverages class-means as reliable feature anchors and aligns the similarity distributions of unlabeled data across sequential tasks using a KL-divergence objective, thus extending knowledge distillation to unlabeled domains [2508.05316].

## 1. Motivation and Role in Memory Stability

SSCL frameworks confront the dual challenge of learning from sequentially arriving tasks and exploiting both labeled and abundant unlabeled data. While most prior approaches focus distillation strategies exclusively on labeled exemplars, the large pool of unlabeled data remains unregulated; as the model evolves, its feature representations for these unlabeled points can drift, exacerbating forgetting. CUD addresses this deficiency by “anchoring” each unlabeled example to a consistent set of class means—vectors computed only from labeled data, and therefore, more resistant to error accumulation across tasks. The similarity profile of each unlabeled feature to these class means (computed via cosine similarity) is used as the distilled target. By aligning these similarity distributions from the frozen previous model to the current one, memory stability is maintained for unlabeled data, thereby curtailing catastrophic forgetting while not requiring any extra supervision or replay [2508.05316, Sec. 3.4].

## 2. Mathematical Formalism

The mechanism underpinning CUD proceeds as follows:

1. **Class Mean Computation**  
   For each class $c$ observed up to task $t$, a set of labeled examples $C^{t,(c)}$ is maintained. Features are extracted via a projection head and L₂-normalized:
   $$
   f_x^t = P^t(F^t(x)),
   $$
   and the class mean is computed as:
   $$
   \mu_{C^{t,(c)}} = \frac{1}{|C^{t,(c)}|} \sum_{x \in C^{t,(c)}} f_x^t.
   $$
   The means are stacked into $M^t \in \mathbb{R}^{d \times k}$.

2. **CUD Loss for Unlabeled Data**  
   For each unlabeled sample $x_u^t \in D_u^t$:
   - Evaluate similarities, normalized by temperature $\xi$ (default $0.1$), to $M^t$ for both the current (“student”) and the frozen previous (“teacher”) model:
     $$
     s_{\text{curr}} = S(f_{x_u}^t, M^t)/\xi,\quad s_{\text{prev}} = S(f_{x_u}^{t-1}, M^t)/\xi
     $$
     where $S(\cdot,\cdot)$ denotes row-wise cosine similarity.
   - The CUD objective is applied as:
     $$
     \mathcal{L}_{\text{cud}}(D_u^t) = \mathbb{E}_{x_u^t \sim D_u^t}\left[ \operatorname{KL}(\operatorname{softmax}(s_{\text{curr}}) \parallel \operatorname{softmax}(s_{\text{prev}})) \right].
     $$
   - The term is weighted by $\lambda_{\text{cud}}$ (default $1.0$), and standard feature normalization is applied throughout [2508.05316, Eq. 9, Sec. 3.4].

## 3. Training Procedure and Pseudocode Integration

CUD operates in tandem with Feature Space Reservation (FSR) and Divide-and-Conquer Pseudo-labeling (DCP) within the broader USP SSCL pipeline. The following outlines the integration:

1. Initialize model with previous parameters, build ETF class prototypes, and compute class means $M^t$ from current labeled data.
2. Within each training epoch and batch:
   - Forward features for labeled ($f_l$), unlabeled ($f_u$), and exemplar data ($f_e$)
   - Compute supervised cross-entropy ($L_{\text{sup}}$), standard CL-distillation ($L_{\text{cl}}$ on exemplars), FSR loss ($L_{\text{fsr}}$), and unsupervised loss using DCP ($L_{\text{uns}}′$)
   - Compute CUD loss ($L_{\text{cud}}$) by aligning similarity distributions for $f_u$ with previous features $f_u^{\text{old}}$ via KL-divergence, as described above
   - Update total loss as the weighted sum and backpropagate
3. Recompute $M^t$ each epoch [2508.05316, Sec. 3.4 and Appendix A.1].

A high-level pseudocode excerpt:

```python
# Given: features f_u, f_u_old; class-mean matrix M; temperature xi
logits_curr = f_u @ M.t() / xi
with torch.no_grad():
    logits_prev = f_u_old @ M.t() / xi
p_prev = logits_prev.softmax(dim=1)
log_p_curr = logits_curr.log_softmax(dim=1)
loss_cud = F.kl_div(log_p_curr, p_prev, reduction='batchmean')
loss += lambda_cud * loss_cud
```

Best practices include detaching previous features/logits, always re-normalizing features, and recomputing $M^t$ periodically [2508.05316, Sec. 6].

## 4. Theoretical Insights and Intuition

CUD leverages the property that class means, being inferred from labeled exemplars, exhibit substantial resistance to model drift and label noise compared to the neural classifier’s weight vectors under continual learning. By compelling current model features for unlabeled data to preserve their similarity relations to these stable class-means—mirroring the previous model’s relational structure—the framework effectively distills “relational knowledge” about the data manifold. This mechanism is supported by known robustness properties of prototype-based approaches (such as nearest class mean classification) in continual settings. While formal proofs are not provided, the hypothesis is that this prototype-anchored, relational distillation constrains feature space drift in a way that ordinary teacher-student logit matching does not [2508.05316, Sec. 3.4].

## 5. Empirical Validation and Quantitative Effects

Ablation and comparative analyses on standard SSCL benchmarks quantify the contributions of CUD:

- On CIFAR-10 (30 labels/class, 5 tasks), removing CUD (“wo. L_cud”) decrements average accuracy by approximately $1.7$ percentage points (from $81.43\%$ to $\sim79.7\%$), while DCP alone yields the largest single gain and CUD confers an additional $1$–$1.5$ point improvement [2508.05316, Fig. 4(a)].
- On CIFAR-100 (25 labels/class, 10 tasks), replacing CUD with ordinary naive logit distillation leads to significantly worse results (average $53.91\%$ and last accuracy $37.97\%$) versus CUD’s $54.36\%$/$38.25\%$ [2508.05316, Sec. 4.2].
- Sensitivity evaluation shows that varying $\lambda_{\text{cud}}$ in $[0.1, 2.0]$ produces less than $2$ percentage points variation around the default, attesting to CUD’s hyperparameter robustness [2508.05316, Fig. 8].
- Across all benchmarks, CUD generates gains in average and “last” accuracy within $0.5$–$2.0$ percentage points, reflecting consistent anti-forgetting benefit [2508.05316].

## 6. Practical Implementation Considerations

CUD is implemented in PyTorch as an additional loss component added to the total SSCL objective. All previous model computations are wrapped in `torch.no_grad()`, class mean matrices are recomputed at least once per epoch, and feature vectors are always L₂-normalized post-operation. No extra storage for prototypes beyond $M^t$ is required, and exemplar buffers remain as in standard continual learning pipelines. The reference implementation is publicly available for inspection and experimentation [2508.05316].

| Component              | Purpose                                      | Notes                                   |
|------------------------|----------------------------------------------|-----------------------------------------|
| Class-mean matrix $M^t$| Stable anchoring for similarity calculation  | Recomputed per epoch                    |
| CUD loss term          | Distill similarity distributions on unlabeled| Weighted by $\lambda_{\text{cud}}$      |
| Feature normalization  | Stability of cosine similarity               | Enforced after each projection          |

The reliance on class-means as the anchor makes this approach lightweight and amenable to integration into existing SSCL architectures.

## 7. Significance and Context within SSCL

CUD represents an evolution in knowledge distillation for continual learning, extending distillation strategies beyond labeled samples to the dominant, otherwise unregulated, pool of unlabeled data. By leveraging stable class-means, CUD strengthens memory stability without introducing large overheads or necessitating additional storage for pseudo-prototypes. This results in improved balancing of stability and plasticity in realistic semi-supervised continual learning environments. Its integration into the USP framework demonstrably boosts performance over prior art, substantiating the value of prototype-anchored relational distillation strategies [2508.05316].

Source: https://www.emergentmind.com/topics/class-mean-anchored-unlabeled-distillation-cud