Unsupervised Continual Clustering via Forward-Backward Knowledge Distillation

Abstract: Unsupervised Continual Learning (UCL) aims to enable neural networks to learn sequential tasks without labels or access to past data. A major challenge in this setting is Catastrophic Forgetting, where models forget previously learned tasks upon learning new ones. This challenge is amplified in UCL due to the absence of labels to guide learning and memory retention. Existing mitigation strategies, such as knowledge distillation and replay buffers, often raise memory and privacy concerns. Moreover, current UCL methods largely overlook clustering-specific objectives. To fill this gap, we introduce Unsupervised Continual Clustering (UCC) and propose Forward-Backward Knowledge Distillation for Continual Clustering (FBCC). FBCC employs a continual teacher network with a clustering projector and lightweight task-specific students. Through a dual-phase forward-backward distillation process, the teacher learns new clusters while preserving previously discovered cluster structure without storing past data. FBCC represents a pioneering approach to UCC, demonstrating improved clustering performance across sequential tasks. Experiments on four benchmark datasets demonstrate that FBCC consistently outperforms existing continual learning baselines in clustering accuracy while significantly reducing catastrophic forgetting.

• The paper introduces FBCC, which integrates a dual-phase (forward-backward) knowledge distillation to jointly learn representations and clustering assignments.
• It employs a teacher-student framework that uses lightweight, task-specific models and incremental prototype updates to counteract catastrophic forgetting.
• Empirical results demonstrate FBCC outperforms both unsupervised and supervised baselines, achieving 75.28% accuracy and 2.29% forgetting on CIFAR-10.

Unsupervised Continual Clustering via Forward-Backward Knowledge Distillation: Technical Summary

Introduction and Motivation

Unsupervised Continual Learning (UCL) addresses the challenge of learning representations from sequentially arriving, unlabeled data without access to past data—a setting where catastrophic forgetting (CF) severely hinders performance. Despite growing literature in UCL, most methods either employ replay buffers (conflicting with data privacy and memory efficiency) or optimize for stable representation learning, relegating clustering to a downstream step rather than an explicit objective. In real-world settings, tasks often correspond to distinct collections or classes, for which cluster assignments must be discovered incrementally and retained over time. This paper introduces the Unsupervised Continual Clustering (UCC) problem, targeting these requirements by integrating clustering objectives into the continual learning paradigm and proposing a novel solution, FBCC, based on forward-backward knowledge distillation.

FBCC Framework

The core innovation of FBCC is the use of a dual-phase distillation mechanism involving a high-capacity teacher network and lightweight, task-specific student networks. The teacher embodies a continual learner, augmented by a clustering projector to explicitly form cluster probability assignments for each task. For every incoming task, a compact student is instantiated to replicate the teacher's representations for that task. The learning process alternates between two forms of knowledge transfer:

• Forward Distillation: During training on a new task, the teacher is regularized by matching its representations against those of frozen students corresponding to earlier tasks. This prevents drift in the latent space corresponding to prior knowledge, thereby suppressing forgetting.
• Backward Distillation: After the teacher has learned a given task, the respective student is trained to mimic the teacher, capturing its task-specific representation and structural information to assist future knowledge retention.

Figure 1: The FBCC architecture employs a teacher–student paradigm, with forward and backward distillation to preserve and transfer task-specific clustering knowledge across tasks.

Prototypes representing discovered clusters from previous tasks are updated incrementally and included in contrastive losses to maintain separation of cluster assignments acquired over the learning trajectory. To address the memory-accuracy tradeoff, FBCC maintains only a limited set of recent student models, defined by hyperparameter $M$, balancing model footprint with anti-forgetting efficacy.

Empirical Analysis

FBCC's effectiveness is evaluated on four vision benchmarks: CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet100. Each dataset is partitioned into multiple tasks, each introducing new, previously unseen clusters. Two primary metrics summarize performance:

• Average Clustering Accuracy ($\overline{\text{ACC}}$): Measures the quality of discovered clusters over all tasks at the end of training.
• Average Forgetting ($\overline{\text{F}}$): Quantifies degradation in cluster assignments for earlier tasks as new tasks are learned.

FBCC is benchmarked against both unsupervised (CCL, STAM, LUMP, CaSSLe, POCON) and supervised continual learning baselines (Co$^2$L, OCD-Net). Notably, it outperforms all baselines in both accuracy and forgetting, despite using substantially fewer parameters compared to models like CaSSLe. Especially on CIFAR-10, FBCC reaches $75.28\%$ $\overline{\text{ACC}}$ and $2.29\%$ $\overline{\text{F}}$, well above the best unsupervised alternatives.

Figure 2: FBCC’s average clustering accuracy and forgetting across tasks as a function of the student pool size $M$ (number of retained student networks per forward distillation phase) exhibit a clear tradeoff, with diminishing returns for large $M$.

An ablation study demonstrates the necessity of (i) maintaining a pool of student models rather than a single one as in CaSSLe, (ii) integrating task prototypes, and (iii) employing explicit knowledge distillation objectives. Removing any of these elements significantly increases forgetting and reduces clustering accuracy.

Robustness, Scalability, and Efficiency

FBCC exhibits robustness to various deployment scenarios:

• Imbalanced Data: Experiments with skewed sample distributions across tasks indicate that the multi-student approach retains task knowledge more effectively over time, even with limited labels for early tasks. Students efficiently mimic teacher representations with only minor accuracy degradation.
• Heterogeneous Tasks: When task sizes and cluster counts vary, FBCC sustains performance, significantly reducing the forgetting effect relative to CaSSLe.
• Semantic Proximity: Grouping tasks by semantically proximal classes (e.g., using CIFAR-100 superclasses) further improves FBCC's performance, indicating enhanced discriminability when clusters are thematically related.

Figure 3: FBCC clustering efficacy remains resilient under highly imbalanced data regimes, in both accuracy and knowledge retention, due to student model specialization and effective backward distillation.

FBCC maintains competitive compute and model size overheads. Its use of lightweight students (e.g., SqueezeNet 1.1) keeps memory usage lower than replay buffer-based or monolithic approaches, while its teacher-student architecture scales efficiently to large, non-stationary datasets.

Theoretical and Practical Implications

The explicit integration of clustering into the continual learning process, combined with FBCC's forward-backward knowledge distillation, sets a new technical standard for UCL settings where cluster preservation—not merely representation stability—is required. Forward-backward distillation leverages modular memory (specialized students) and prototype-based structure to realistic, privacy-sensitive, or resource-constrained streaming scenarios where storing raw data is infeasible. Importantly, FBCC demonstrates that representation drift and catastrophic forgetting in unsupervised settings are best tackled by disentangling task-specific knowledge and persistently encoding it via small student surrogates rather than global constraints or replay.

These insights inform the design of efficient continual learners for domains such as dynamic anomaly detection, exploratory clustering in unknown environments, and privacy-aware federated learning, where both memory and retrainability are critical constraints.

Conclusion

FBCC advances the state-of-the-art in unsupervised continual clustering by jointly optimizing for representation and clustering objectives, employing a dual-phase distillation algorithm that provides robust, memory-efficient mitigation of catastrophic forgetting. Its empirical dominance and flexible architecture suggest fertile ground for further research on modular memory approaches, continual structure learning, and privacy-preserving unsupervised lifelong learning (2606.07474). Future directions could investigate hierarchical student deployment, adaptive student pool management, and extensions to other unsupervised structure-discovery problems (e.g., manifold learning, density estimation) in continual learning contexts.