Forgetting, plasticity, and co-observation: a third facet of continual learning

Abstract: Efficient continual learning remains a fundamental challenge for deep neural networks. While catastrophic forgetting and loss of plasticity are widely considered the primary obstacles to overcome, we show that these two issues cannot fully explain the performance gap between naive sequential training and offline joint training. In this paper, we highlight data co-observation as a distinct factor influencing continual learning performance. By decoupling the constraints of separate data access from stability and plasticity, we systematically investigate the representational benefits gained by observing training data together. Empirically, we demonstrate a consistent performance difference between joint and separate training across both supervised and self-supervised paradigms in generic data-incremental "chunking" scenarios, whilst mitigating forgetting and controlling for plasticity. Our findings indicate that simultaneous observation of training data (co-observation) yields benefits to the learner's generalization that extend well beyond mere knowledge retention, and that this effect does not require a specific continual distribution shift. Furthermore, we contextualize prominent continual learning mechanisms through this lens: while distillation-based approaches act only as effective knowledge retention mechanisms, our results suggest that the empirical success of memory replay goes beyond the mitigation of forgetting, actively reintroducing the benefits of data co-observation into the learning process.

• The paper introduces co-observation as a distinct continual-learning limitation, showing that even perfect retention and plasticity cannot match joint training when data partitions are never seen together.
• A four-way linear-probing diagnostic isolates co-observation from forgetting, with incremental joint training reaching 74.29% versus 70.39% for checkpoint ensembles and 64.64% for naive sequential training on CIFAR-100.
• The results show that replay can restore cross-chunk feature learning while distillation mainly preserves existing representations, reframing replay as both a forgetting-control and co-observation mechanism.

The paper argues that the standard stability-plasticity framing of continual learning (CL) is incomplete: even a hypothetical learner with perfect stability and perfect plasticity would still underperform offline joint training because it never observes data from different partitions simultaneously. The authors call this the co-observation effect and provide a diagnostic methodology that isolates it from catastrophic forgetting, demonstrating it across supervised and self-supervised continual pre-training in data-incremental "chunking" scenarios (2608.18803).

Conceptual argument

The authors partition a data distribution into sequential chunks $\mathcal{D}_1, \dots, \mathcal{D}_T$ and posit a set of shared features $Z_{\text{shared}}$ that improve prediction across the joint distribution. When optimizing only on $\mathcal{D}_1$, neural networks favor simpler solutions (simplicity bias) and settle for task-local shortcuts $Z_{\text{local}}$; $Z_{\text{shared}}$ is never required to minimize loss and becomes unidentifiable from the resulting parameters. An ideal continual learner with zero forgetting and full plasticity would faithfully retain $\theta_1$, but those parameters simply do not encode the discarded shared features, and the same shortcut dynamics repeat on $\mathcal{D}_2$. Discovering cross-distribution synergies, the argument goes, fundamentally requires updating features based on their simultaneous performance across distributions — a constraint that separate training cannot supply. This connects to multi-task learning theory, where joint optimization provides a mutually beneficial inductive bias and reduced overfitting risk of shared parameters (2608.18803).

Diagnostic methodology

The core experimental device is a four-way comparison evaluated by linear probing (LP) on frozen representations, which removes confounds from a continually shifting classifier head:

Baseline	Forgetting	Co-observation	Plasticity
Joint (offline)	mitigated	present	present
Incremental joint	mitigated	present	reduced
Ensemble (checkpoint concat.)	mitigated	absent	reduced
Naive sequential	present	absent	reduced

After each chunk, a frozen copy of the feature extractor is stored; at evaluation, representations from all $T$ checkpoints are concatenated (optionally PCA-projected back to the original dimension $d$ to control for capacity) and probed. Because the ensemble preserves every historical representation, forgetting is eliminated by design, so the residual gap between ensemble and incremental joint training is attributed to the absence of co-observation. The authors verify empirically that train-set accuracy on past chunks never declines once trained, confirming zero forgetting, and ablate the forward/backward-transfer confound by rebuilding the ensemble from distillation-enhanced (LwF/CaSSLe) checkpoints — which yields at best marginal improvements and does not close the gap (2608.18803).

Proof of principle

A controlled two-task MNIST experiment demonstrates the effect explicitly. Task 1 is odd/even classification on clean digits; Task 2 is small/large classification on Gaussian-noise-corrupted digits; evaluation is odd/even on noisy digits, which requires combining the decision boundary from Task 1 with noise robustness from Task 2. The ensemble outperforms naive sequential training (isolating forgetting), but incremental joint training outperforms the ensemble: noise features transfer to the odd/even evaluation task only when noisy and clean data are observed simultaneously, even though they are associated with different tasks (2608.18803).

Co-observation in chunking scenarios

The main experiments use the "chunking" protocol, where the training set is randomly and uniformly split into four identically distributed chunks — a deliberately hard setting because it removes macroscopic distribution shift and class imbalance. All methods receive an identical, fixed iteration budget calibrated to the joint-training schedule, so gains cannot be attributed to extended optimization.

Supervised learning (ResNet-18, cross-entropy on CIFAR-100 and ImageNet-100): the ensemble substantially improves over naive sequential training, PCA reduction leaves it essentially unchanged, and incremental joint training consistently surpasses it. On CIFAR-100, final-chunk LP accuracy reaches 74.29% for incremental joint versus 70.39% for the ensemble and 64.64% for naive, against a 75.18% offline joint target; on IN-100 the corresponding figures are 82.75% (incremental joint), 80.55% (ensemble), 78.27% (naive), and 83.68% (offline joint). Extending to 8- and 12-chunk sequences on IN-100 preserves the ensemble-to-incremental-joint gap, indicating the effect is robust to fragmentation granularity (2608.18803).

Self-supervised learning: with Barlow Twins (ResNet-18) and I-JEPA (ViT-B/16) on IN-100, the same ordering holds — the ensemble consistently falls short of incremental joint training (e.g., Barlow Twins: 73.50% vs. 71.25%; I-JEPA: 62.64% vs. 59.82%). Notably, the effect persists despite SSL's reputation for learning diverse, task-agnostic features that are comparatively robust to forgetting, establishing the co-observation gap as a property of fragmented optimization itself rather than of any particular objective. One caveat: PCA reduction of the ensemble helps Barlow Twins slightly but hurts I-JEPA, so capacity control interacts with the SSL objective (2608.18803).

Reinterpreting replay and distillation

The comparison against CL mechanisms yields the paper's most consequential reinterpretations:

• Experience replay is conventionally framed as a forgetting-mitigation tool. Here, replay with a small buffer (~3% exemplars) fails to consistently match even the ensemble at representation level; ~15% roughly matches it; and a 30% buffer pushes past the ensemble, closing roughly half the gap to incremental joint training (e.g., 81.27% vs. ensemble 80.55% and incremental joint 82.75% on IN-100 supervised). Since training budgets are strictly controlled, these gains reflect replay's second function: reconstructing minibatch-level co-observation, enabling cross-distribution features that perfect retention alone cannot recover. The implication is that replay's empirical success has been systematically misattributed, and that substantial buffer sizes are needed before co-observation benefits materialize.
• Knowledge distillation (LwF supervised; CaSSLe for Barlow Twins) tracks the ensemble closely (e.g., LwF at 79.93% vs. ensemble 80.55% on IN-100). The authors stress that distillation is not failing to prevent forgetting — it matches the perfect-retention baseline — but that distilling from an "incomplete" previous representation cannot restore the co-observation conditions needed for joint-level generalization. Distillation is thus characterized as a pure retention mechanism.

For pre-trained-model CL, the authors note that frozen-backbone prototype methods and conservative fine-tuning bypass or overpower the effect, whereas continual pre-training is where co-observation should matter most; they propose that replay's necessity in LLM continual pre-training may partly reflect co-observation restoration rather than forgetting prevention alone, and offer their diagnostic framework as a starting point for disentangling the two at scale (2608.18803).

Limitations and open questions

The paper concedes three limitations. The ensemble's capacity control relies on post-hoc PCA rather than an ideal single model with matched parameter count, making it an analytical proxy rather than a native perfect-retention learner. Uniform optimization schedules are chosen for comparability, not maximal performance, so other continual optimizers might shift absolute numbers (though the authors expect the trend to hold). Finally, experiments are restricted to moderate-scale vision benchmarks; whether billion-parameter over-parameterization mitigates or exacerbates the need for co-observation remains open, as does the concrete question of how to synthesize cross-distribution synergies in compute-bounded continual learning.

Conclusion

The paper establishes data co-observation as a third, distinct facet of continual learning alongside forgetting and plasticity, supported by a controlled diagnostic that removes forgetting by construction and evaluates at representation level. The effect emerges generically in identically distributed chunking scenarios, persists across supervised and self-supervised objectives and sequence lengths, and reframes replay as a co-observation-restoring mechanism while demoting distillation to retention only. Its central claim — that perfect stability and plasticity still leave a representational ceiling below joint training — directly challenges the implicit evaluation assumption underlying much of the CL literature.