Re-Evaluating Continual Learning with Few-Shot Adaptation

Abstract: Continual learning methods aim to maximize the stability and plasticity of machine learning models that are trained on a sequence of tasks. The standard measure of stability (i.e., forgetting) is the 0-shot performance of a model on previously learned tasks, and plasticity, the performance on the most recently learned task. However, 0-shot evaluation does not fully measure a model or method's ability to retain learned information or adapt quickly to new information, as it requires perfect recall across multiple tasks. In this paper, we propose few-shot evaluation as a more comprehensive assessment of the stability and plasticity of a continual learning system. We conduct a fine-grained assessment on task sequences for continual image classification and find that this paradigm produces novel insights into the performance of popular continual learning strategies. Through few-shot evaluation with a novel metric -- per-shot plasticity -- we show that adding `foresight' to continual learning methods via the meta-learning of a short sequence of future tasks induces learning-to-learn behavior over the task sequence.

• The paper introduces a few-shot adaptation protocol that reveals hidden dynamics in continual learning by addressing the shortcomings of standard 0-shot evaluation.
• It presents SAUCE, a novel metric quantifying per-shot plasticity to measure rapid recovery and adaptation rates, challenging the view of catastrophic forgetting.
• The foresight meta-learning algorithm enhances both backward and forward transfer, demonstrating improved learning-to-learn behavior across diverse CL benchmarks.

Re-Evaluating Continual Learning with Few-Shot Adaptation: Summary and Analysis

Introduction

This paper, "Re-Evaluating Continual Learning with Few-Shot Adaptation" (2606.03843), systematically challenges standard evaluation protocols in continual learning (CL), demonstrating that the prevailing emphasis on 0-shot performance (i.e., immediate recall without adaptation) provides an incomplete and potentially misleading assessment of stability and plasticity in CL systems. Instead, the authors propose a few-shot adaptation protocol and introduce a novel metric, per-shot plasticity (SAUCE), that better quantifies both the recoverability of prior knowledge and rapid adaptation to new tasks. They also present a foresight meta-learning algorithm, which meta-initializes models based on a lookahead over future tasks to explicitly optimize few-shot adaptability, and dissect its effect on a suite of established CL benchmarks and methods.

Limitations of Conventional CL Evaluation

Standard evaluation in CL typically quantifies:

• Stability: Resistance to catastrophic forgetting, measured by 0-shot backward transfer.
• Plasticity: Efficacy on current or the most recently trained task, measured 0-shot.

However, the authors identify two critical deficiencies:

1. 0-shot backward evaluation conflates encoding with accessibility: Poor 0-shot performance does not necessarily indicate catastrophic parameter drift. As observed in natural learners, models might rapidly reacquire “forgotten” behaviors with minimal cues—a phenomenon akin to the psychological ‘relearning effect’. This crucial process is unmeasured by standard metrics.
2. Plasticity measured solely by post-training accuracy is uninformative: When models are sufficiently expressive, plasticity saturates and does not capture the efficiency of within-task adaptation, especially under a limited supervision regime.

The authors advocate for a few-shot evaluation framework in which each post-task checkpoint is subjected to adaptation with $k$ labeled examples from every prior and novel task, providing both granular and task-sequenced insights into the recoverability and rapid adaptation behaviors of CL algorithms.

Figure 1: Illustration of the proposed few-shot adaptation and evaluation protocol in continual learning, including evaluation after adaptation and foresight meta-learning for optimizing future few-shot performance.

Empirical Analysis: Few-Shot Protocol Uncovers Hidden Dynamics

The evaluation comprehensively scrutinizes ER, DER++, EWC, AGEM, MER, and a vanilla SGD baseline, across a range of vision CL benchmarks (class- and domain-incremental splits of MNIST and CIFAR-100 with varying degrees of task overlap and independence).

Recoverability of Knowledge and the Myth of Catastrophic Forgetting

Under 0-shot evaluation, methods incorporating replay (ER, DER++) maintain strong backward accuracy, whereas regularization (EWC, AGEM) and vanilla SGD display apparent catastrophic forgetting. However, when allowed just 5 adaptation examples (5-shot), all methods—including SGD—recover prior task accuracy to near-optimal levels.

Figure 2: With 0-shot evaluation, non-replay methods appear to catastrophically forget prior tasks, but with as few as 5 adaptation samples, backward accuracy improves substantially, exposing the non-catastrophic nature of so-called forgetting for many approaches.

This result indicates that the parameter trajectories of even simple fine-tuning baselines retain significant information, inaccessible in a zero-shot regime but eminently recoverable by contextual adaptation. The claim here is boldly at odds with common interpretations of 0-shot forgetting as indicative of irretrievable knowledge loss. Thus, practical deployment may need only store a small buffer of adaptation samples, rather than incur the cost of continuous replay during training.

Disentangling Plasticity via Few-Shot Forward Transfer

On current and forward tasks, standard (0-shot) measures often indicate saturated and indistinguishable plasticity across methods. However, evaluating 10-shot adaptation exposes substantial differentiation: EWC rapidly loses plasticity as sequences lengthen, AGEM’s adaptation depends critically on cross-task structure (being highly effective when tasks are independent), replay methods are sample-inefficient for rapid adaptation, and SGD consistently exhibits high plasticity due to the absence of restrictive regularization biases.

Figure 3: Comparison between 0-shot plasticity, which is largely saturated, and 10-shot forward adaptation, which sharply differentiates methods, particularly on extended task sequences.

Longitudinal Analysis: Learning-to-Learn Induction

By tracking per-checkpoint few-shot performance across long sequences, the paper establishes that most existing methods fail to exhibit improvement in adaptation rate (“learning-to-learn” capabilities) as learning progresses; in some settings, plasticity even decays.

Figure 4: Ten-shot adaptation curves over task sequences show static or deteriorating plasticity for most methods in the absence of explicit meta-learning.

Foresight Meta-Learning: Optimizing Few-Shot Adaptivity

To induce explicit improvements in few-shot adaptation over the learning trajectory, the paper introduces a foresight meta-learning approach using an MAML-style inner loop that meta-initializes on a short lookahead window of future tasks. Unlike prior methods that meta-learn from previously seen data (hindsight), this approach utilizes as-of-yet unseen (future) samples, directly optimizing the initialization for adaptability to upcoming tasks, constrained by computational practicalities (e.g., $L \leq 3$ lookahead tasks).

The empirical impact of foresight meta-learning is clear:

• Forward few-shot transfer improves in almost every scenario.
• Backward transfer often also benefits, despite no direct optimization for backward adaptation.

Figure 5: Augmentation of standard CL methods with foresight meta-learning improves both 5-shot backward and forward transfer, particularly on sequences with cross-task structure.

Per-Shot Plasticity: The SAUCE Metric

The authors introduce SAUCE (Scaled Area Under the adaptation CurvE), a normalized measure reflecting the rate of adaptation—how quickly a model improves given additional adaptation samples. Unlike aggregate k-shot accuracy or the area-under-learning-curve (LCA), SAUCE isolates dynamic improvement rather than conflating it with ceiling accuracy levels.

Figure 6: The SAUCE metric measures cumulative adaptation (regret) and normalizes to isolate true adaptation rapidity, not just higher ceiling performance.

Key findings using SAUCE:

• Replay methods maintain high ultimate accuracy but are less sample-efficient (slower adaptation) compared to SGD and regularization baselines.
• Augmenting with foresight meta-learning consistently decreases (improves) per-shot plasticity (SAUCE) across a range of methods, especially for AGEM.
• Longitudinally, only methods with foresight meta-learning demonstrate a monotonic increase in plasticity, evidencing learning-to-learn behavior; other methods are static or degrade.

Figure 7: Foresight meta-learning consistently reduces SAUCE (improves rapid adaptation), especially in low-task-overlap regimes.

Figure 8: Only models equipped with foresight meta-learning acquire increased adaptation rates across long task sequences—demonstrating the emergence of learning-to-learn.

Implications and Future Directions

This work compels a re-thinking of CL evaluation—advocating rigorous few-shot protocols and warning against overinterpretation of standard 0-shot forgetting. Practically, the findings suggest resource- and sample-efficient deployment strategies: test-time adaption with a tiny buffer may substitute for costly, large-scale replay during training. Replay methods, while ensuring high final accuracy, are not always optimal for settings with severe adaptation constraints.

The foresight meta-learning framework bridges CL and meta-learning, optimizing not for mere reduction in forgetting but for holistic, rapid adaptability—crucial both for real-world continual deployment and as a mechanistic model of learning-to-learn in neural networks.

Theoretically, the methodology generalizes beyond vision and opens a path for CL protocols focusing on contextual- or in-context adaptation. The authors speculate on future extensions, particularly the integration of in-context learning for foundation models and advanced ways for meta-learning few-shot adaptation rather than direct task accuracy—an essential trajectory as foundation models become the backbone of life-long learning AI.

Conclusion

This paper provides a rigorous, quantitative, and critical re-examination of continual learning, revealing that so-called “catastrophic forgetting” is often a failure of immediate recall, not content obliteration. It documents that with just a few adaptation examples, even naïve baselines recover prior knowledge effectively. The introduction of SAUCE as a metric and a foresight meta-learning framework for explicitly meta-optimizing adaptation rate marks an important methodological advance for future research. The findings urge the adoption of few-shot protocols and dynamic adaptation metrics for fair, meaningful evaluation and point to a substantial underexploited capacity for learning-to-learn in neural architectures.