The Gentle Collapse: Distributional Metrics for Continual Learning

Abstract: Accuracy degradation is the standard metric for Catastrophic Forgetting (CF), however, it records only whether forgetting occurred or not. It saturates at the extremes and collapses discretely at task boundaries, hiding the internal structure of what is being forgotten. We introduce six softmax-derived metrics spanning true-label rank (TLR), predictive confidence, and distributional divergence that characterize forgetting continuously, each normalized to [0, 1] with no modification to training. On CIFAR-100, these metrics carry information where accuracy does not: at 0% accuracy, the Confusion Margin spans an IQR of [0.32, 0.50] across classes that accuracy treats identically. We demonstrate that this richer signal is actionable in mitigating catastrophic forgetting. Per-sample metric scores used as loss weights reduce forgetting by 1.3 percentage points over uniform experience replay (ER) on CIFAR-100. Furthermore, the slope of a metric over a small window provides a stable sampling criterion: at a small-window size (e.g. 3 epochs), accuracy-trend degrades to 34.79% (std. = 2.32) while log-TLR achieves 41.07% (std. = 0.57). This gap is structural since reliable small-window trend estimation requires a continuous signal. On TinyImageNet, log-TLR trend sampling reduces forgetting by 7.7 percentage points over the ER baseline.

• The paper introduces six softmax-derived metrics that capture gradual forgetting in continual learning beyond binary accuracy measurement.
• It demonstrates that distributional metrics such as TLR, Log-TLR, and CTL provide stable, continuous signals for diagnosing and mitigating catastrophic forgetting.
• Empirical results on CIFAR-100 and TinyImageNet show that metric-based replay prioritization significantly improves performance compared to traditional accuracy metrics.

Distributional Metrics for Continual Learning: A Detailed Assessment

Introduction

"The Gentle Collapse: Distributional Metrics for Continual Learning" (2606.25165) provides a critical examination of the limitations inherent in accuracy-based evaluation for catastrophic forgetting (CF) in continual learning (CL) and introduces six normalized, softmax-derived metrics for richer, continuous characterization of forgetting. This work systematically demonstrates that reliance on accuracy, a discrete and binary metric, masks the nuanced dynamics of knowledge erosion across tasks and classes, effectively constraining both the diagnosis and mitigation of CF within neural networks.

Limitations of Accuracy-Based Forgetting Metrics

Accuracy is binary and aligned with the decision boundary, offering minimal interpretability concerning the model's residual knowledge or the degree of forgetting prior to and after misclassification. The authors underscore that accuracy collapses abruptly at task boundaries, eliminating gradation and yielding little actionable signal for many downstream interventions.

Figure 1: Mean $\pm$ std across Task-1 classes for accuracy, confusion margin, and true-label rank centered on the Task 1$\rightarrow$2 boundary; accuracy collapses sharply, while continuous metrics reveal gradual forgetting.

The figure illustrates the abrupt collapse characteristic of accuracy and, by contrast, the smooth decay of distributional metrics, which preserve information about intermediate forgetting states.

Distributional Metrics: Definitions and Taxonomy

The six proposed metrics fall into three categories:

• True-Label Rank Family:
  • TLR: Linear mapping of the true class rank in the predicted list.
  • Log-TLR: Logarithmic sensitivity, reducing the impact of severe misranks.
  • Reciprocal-TLR: Weighted sensitivity toward top ranks, most stringent for near-correct predictions.
• Confidence and Confusion Metrics:
  • True-Label Confidence (CTL): Average predicted softmax for the correct class (proper scoring rule).
  • Confusion Margin (CM): Difference between top predicted probability and correct label, quantifying proximity to misclassification.
• Distribution Matching:
  • Normalized KL (NKL): KL divergence to one-hot, normalized to $[0,1]$; quantifies global misalignment of predicted probability mass.

All metrics are distributed in $[0,1]$ and incur no additional computational cost or modifications to model architecture or inference flow.

Empirical Analysis of CF Dynamics

Graded Dynamics Beyond Accuracy

Distributional metrics reveal the trajectory of forgetting in a continuous fashion. During task transitions, they expose persistent knowledge—or rapid loss thereof—for individual classes even when accuracy is saturated at zero or one. The paper shows that, at $0\%$ class accuracy, CM can span an interquartile range (IQR) of $[0.32, 0.50]$, meaning some classes are on the cusp of being recovered whereas others are fully lost; accuracy provides no such stratification.

Figure 2: At fixed accuracy levels ($\pm0.5\%$ windows), metrics like CM and TLR show wide distributions, indicating residual information where accuracy is uninformative.

Temporal Smoothness and Intervention Feasibility

Accuracy exhibits extreme jumps at task boundaries (epoch-to-epoch absolute differences $>0.35$ are noted), which destabilizes short-window trend estimation. Continuous metrics (CM, TLR, Log-TLR) are substantially less volatile and thus support robust temporal analysis.

Figure 3: Epoch-to-epoch change is extreme for accuracy, particularly at task boundaries, whereas continuous metrics display moderate, stable changes.

Actionable Use of Distributional Signals

Per-Sample Loss Weighting

Per-sample scores derived from distributional metrics can modulate replay buffer gradient contributions, with the most-forgotten samples exerting stronger influence on parameter updates. On CIFAR-100, weighting by Log-TLR reduced forgetting by 1.3 percentage points over uniform experience replay (ER), and similar trends were observed for TinyImageNet.

Replay Prioritization via Trend Sampling

The slope of distributional metrics over short windows forms a stable prioritization signal, unlike accuracy's noise-dominated, binary trend. With a window size $w=3$ epochs on CIFAR-100, accuracy-based trend sampling yields $34.79\pm2.32\%$ accuracy—over six percentage points below any continuous metric under the same protocol.

Figure 4: Final accuracy on Sequential CIFAR-100 versus trend window size; accuracy-based trend estimation fails for short windows while continuous metrics remain robust.

On TinyImageNet at $\rightarrow$0, Log-TLR trend-based sampling decreases forgetting by 7.7 percentage points compared to the baseline, demonstrating both higher efficacy and lower variance than loss or other metrics. This validates the structural advantage of continuous metrics for rapid, online interventions, especially in environments where access to extended historical data is infeasible.

Complementarity, Limitations, and Future Directions

The study highlights that no single metric is universally optimal: rank-based metrics provide signal at severe forgetting, while confidence-based measures excel in mid-forgetting regimes. This complementarity opens avenues for hybrid metric schemes tailored to specific buffer management or prioritization strategies. Notably, the approach is orthogonal to method class (replay, regularization, architecture) and thus widely applicable.

The authors propose extending fine-grained, metric-aware interventions to advanced replay algorithms, such as DER++ and iCaRL, where surjective interactions between logit-matching and metric-weighting require further investigation. There is potential synergy between distributional-metric evaluation and uncertainty-aware or distillation-based continual learning methods as well.

Conclusion

This paper systematically demonstrates that the granularity of the evaluation signal in continual learning is directly tied to the quality and feasibility of interventions. By introducing continuous, softmax-derived metrics, the authors provide actionable tools for diagnosing, tracking, and mitigating catastrophic forgetting at a finer scale than possible with accuracy alone. These metrics enable both more stable and more effective prioritization in replay-based CL, supporting nuanced, temporally responsive strategies without architectural overhead. Their adoption is likely to catalyze further progress in continual learning protocol design and evaluation and to warrant integration into new and existing CL algorithms targeting both theoretical understanding and practical robustness.