---
title: 'Continual Learning: Definition, Applications, and Evaluation'
url: https://www.emergentmind.com/topics/continual-learning
type: topic
---

# Continual Learning: Definition, Applications, and Evaluation

Continual learning (CL), also called lifelong or incremental learning, is the study of models and agents that acquire knowledge from sequentially arriving, potentially non-stationary data while retaining useful capabilities learned previously. Unlike batch learning, in which training data are assumed to be jointly available, CL typically restricts access to earlier data and requires adaptation under memory, computation, privacy, energy, or communication constraints. Its central technical problem is the stability–plasticity dilemma: plasticity permits acquisition of new information, whereas stability preserves prior knowledge. Excessive plasticity produces catastrophic forgetting; excessive stability produces intransigence, in which the model fails to learn new tasks [2311.11908].

## 1. Problem formulation and settings

A continual-learning stream can be represented as a sequence of datasets or tasks,

$$
\mathcal D_1,\mathcal D_2,\ldots,\mathcal D_T,
$$

where task-specific distributions may differ across time. At step $t$, the learner has access to current data $\mathcal D_t$ and to whatever summary of earlier experience has been deliberately retained. Raw historical data may be permanently discarded because of storage, privacy, legal, computational, or operational constraints [1902.06494].

The learner is expected to perform well on the current task and on previously encountered tasks. In a performance matrix, $a_{t,s}$ denotes performance on task $s$ after learning through task $t$. The diagonal measures current-task learning, the lower-triangular entries measure retention, and the upper-triangular entries can measure forward transfer. This formulation distinguishes adaptation from retention and transfer rather than reducing CL to final-task accuracy.

Three commonly studied settings are:

- **Task-incremental learning**: task identity is available at inference, usually enabling selection of a task-specific output head.
- **Domain-incremental learning**: the label space remains fixed while the input distribution changes.
- **Class-incremental learning**: new classes arrive over time and task identity is unavailable at inference. The model must discriminate both within and across tasks; this is generally the most difficult of the three [2109.11369].

The distinction between task-incremental and task-agnostic evaluation is consequential. A multi-head system can use task identity to select a specialized head, whereas a single-head class-incremental system must infer the correct class among all classes encountered so far. In sequential classification, the latter setting exposes failures that are hidden by task-aware evaluation.

Continual learning also encompasses online or infinite-stream learning, in which examples arrive individually or in small batches without explicit task boundaries. Such streams may be temporally correlated, multimodal, noisy, incomplete, and affected by the learner’s own actions. Autonomous agents must additionally select goals, acquire behaviors, organize knowledge, and determine which experiences are informative [1907.01929].

## 2. Catastrophic forgetting and its causes

Catastrophic forgetting is the degradation of performance on earlier tasks after learning later tasks. In neural networks, updates optimized for a new distribution modify shared parameters that also encode earlier representations and decision boundaries. When old data are unavailable, the optimizer receives no direct signal identifying which changes would damage prior behavior [1810.10612].

Forgetting is not the only source of performance degradation. Continual models may select features that are predictive locally but non-generalizable globally. A **spurious feature** correlates with labels in training data but not in the test distribution because of covariate shift. A **local spurious feature** is predictive within the currently observed task but not across the complete continual sequence. Consequently, a model can show poor final performance even when earlier parameters are frozen and cannot have undergone catastrophic forgetting [2203.01012].

This distinction is important for algorithm design. Preserving every apparently important parameter can preserve an undesirable shortcut. In some cases, beneficial adaptation requires changing a representation that was useful on an earlier but locally misleading task. Thus, forgetting of bad features can improve future generalization, whereas forgetting of invariant features is harmful.

The stability–plasticity trade-off can also be expressed geometrically. Under a local quadratic approximation of old-task losses, a new update $\Delta_t$ that produces zero forgetting must lie in the null space of the aggregate old-task Hessian,

$$
\Delta_t^\intercal \overline{H}_{<t}^{\star}\Delta_t=0.
$$

Flat directions permit learning new tasks with little increase in old loss; sharp directions constrain plasticity. The validity of this analysis depends on task solutions remaining within the neighborhoods where the local approximations are accurate [2407.16611].

A second fundamental issue is limited information capacity. Worst-case complexity results show that one-pass continual learning can require memory growing linearly with the number of tasks, even for improper learners. With multiple passes, multiplicative-weights methods can reduce memory dependence, but the most memory-efficient construction is improper and returns an aggregate function outside the original hypothesis class [2204.10830].

## 3. Principal methodological families

### Replay and rehearsal

Rehearsal stores a bounded subset of previous examples, features, prototypes, or soft labels and mixes them with current data. Replay allows the learner to compare old and new classes directly, reconstruct prior decision boundaries, and revisit old examples after acquiring new representations [2007.00487].

Memory management is therefore central. Candidate strategies include random sampling, reservoir sampling, class-balanced storage, herding-based coresets, uncertainty-based selection, interference-based retrieval, compressed representations, and gradient-aware selection. GEM constrains the current update so that it does not increase loss on stored examples; A-GEM uses an averaged reference gradient.

Generative replay replaces raw examples with samples produced by a learned generative model. The teacher or generator summarizes prior distributions, produces synthetic input-label pairs, and supports training of a new student. After training, the student becomes the next teacher. This avoids retaining raw datasets and separate models for every task, but retention is bounded by generative fidelity. Errors can be propagated and amplified across successive generations [1810.10612].

Generative replay can be marginal or conditional. Marginal generators produce samples without explicit class labels; conditional generators receive a class or task condition and can control class balance more directly. On MNIST and Fashion-MNIST, generative replay can be highly effective, whereas experiments on CIFAR10 show that blurry samples, mode collapse, and accumulated generation errors can make the generator the principal bottleneck [2007.00487].

Manifold Expansion Replay (MaER) extends rehearsal by selecting memory examples that expand the diameter of the feature-space region represented by the buffer. It combines this memory-management strategy with Wasserstein feature distillation between teacher and student representations. The method aims to improve geometric coverage rather than merely preserve class frequencies, although its centroid-based diameter criterion can overemphasize outliers or miss disconnected modes [2310.08038].

### Regularization and Bayesian methods

Regularization methods constrain changes to parameters or functions associated with prior tasks. EWC uses Fisher-information estimates,

$$
\mathcal L(\theta)=\mathcal L_{\mathrm{new}}(\theta)+
\frac{\lambda}{2}\sum_i F_i(\theta_i-\theta_i^{\mathrm{old}})^2.
$$

Synaptic Intelligence accumulates parameter importance from the contribution of parameter changes to loss reduction. Memory Aware Synapses estimates importance from output sensitivity and can operate without labels. Neuron-activation-importance methods instead assign a common importance coefficient to all incoming weights of a neuron, based on activation magnitude normalized by activation variability. This construction seeks to protect coherent feature maps rather than individual weights [2107.12657].

Bayesian continual learning interprets prior-focused methods as using an approximate posterior over parameters as the next task’s prior. VCL, EWC, SI, and related methods constrain movement in parameter space. Likelihood-focused methods instead preserve historical likelihood terms through coresets or generative replay. A hybrid objective combines replay-based likelihood terms with a posterior-based KL regularizer [1902.06494].

The weakness of approximate posterior-as-prior methods is error propagation. A mean-field or otherwise restricted posterior may discard correlations, represent only one mode, become miscalibrated, or fail to encode uncertainty about future input regions. Exact Bayesian updating would not have this problem, but exact posteriors for Bayesian neural networks are generally intractable.

An alternative is to keep the neural representation fixed and update only an exact or analytically tractable statistical state. Sequential Bayesian Meta-Continual Learning meta-trains neural networks to map observations into likelihood contributions for a fixed-dimensional exponential-family posterior. For a diagonal Gaussian posterior,

$$
\Lambda_t=\Lambda_{t-1}+P_t,\qquad
\mu_t=\Lambda_t^{-1}
\left(\Lambda_{t-1}\mu_{t-1}+P_tz_t\right).
$$

Because sufficient-statistic aggregation is commutative, sequential and batch Bayesian updates coincide under the assumed model. The approach avoids parameter overwriting during deployment, although its guarantees depend on the adequacy of the chosen posterior family and its order-insensitive representation [2405.18758].

### Distillation and functional preservation

Knowledge distillation retains the behavior of a previous model by matching outputs, intermediate representations, attention maps, uncertainty, or geometric structure. Learning without Forgetting applies the previous model to current inputs and uses its soft predictions as targets. Distillation avoids storing old images but can be ineffective when current inputs provide little information about old-task behavior. It may also preserve old errors or bias the model toward newly introduced classes [2109.11369].

Functional preservation is particularly important in retrieval, segmentation, detection, and generative modeling, where retaining a parameter vector does not necessarily preserve the behavior relevant to the application. In neural information retrieval, replay and regularization have architecture-dependent effects: replay is especially effective for pretrained transformer rankers, whereas parameter regularization often works better for embedding-based rankers [2308.08378].

### Parameter isolation and architectural methods

Parameter-isolation methods allocate separate parameters, subnetworks, paths, masks, experts, or modules to different tasks. Progressive Networks add task-specific subnetworks; PackNet prunes and freezes weights; HAT learns task masks; Piggyback learns binary masks over a shared network; Expert Gate routes examples to experts; and dynamically expandable networks add capacity as tasks arrive.

These methods can strongly reduce interference, but their costs include model growth, routing complexity, task-identification requirements, and reduced transfer. A fixed-capacity model eventually exhausts free parameters, while strict isolation can prevent useful backward transfer. Continual learning with Echo State Networks illustrates a different form of isolation: the recurrent reservoir remains fixed and only the linear readout is trained. Streaming Linear Discriminant Analysis can then update class statistics online without replay, although replay favored LSTMs over ESNs in the reported experiments [2105.07674].

### Local and global objective approximations

A useful classification cuts across the usual method families. An old-task approximation is **local** when it depends on the historical task solution and is accurate near that point; it is **global** when it does not depend on the parameters obtained after the old task.

EWC and structured Laplace methods are local because their curvature estimates are centered at old task solutions. Experience Replay and GEM are global when their approximations are determined by stored examples independently of subsequent parameter locations. Synaptic Intelligence is classified as global under this criterion because its quadratic estimate is recomputed relative to the current state. iCaRL’s herding-based exemplar selection is local because it depends on the learned representation, whereas random sampling removes that parameter dependence [2407.16611].

Local methods can offer strong stability when task solutions remain close, but may fail under large parameter displacement. Global methods do not require this locality assumption, although their accuracy depends on buffer size, sampling quality, or model bias.

## 4. Evaluation and experimental infrastructure

Continual-learning evaluation must measure current-task adaptation, retention, transfer, resource use, and often uncertainty. Common metrics include average accuracy, forgetting, backward transfer, forward transfer, intransigence, memory, and computation. If $a_{t,s}$ denotes performance on task $s$ after learning through task $t$, average accuracy after $T$ tasks is

$$
A_T=\frac{1}{T}\sum_{s=1}^{T}a_{T,s}.
$$

Forgetting on task $s$ compares its best historical performance with its final performance. Backward transfer measures the effect of later learning on earlier tasks; negative values indicate interference. Forward transfer measures whether previous learning improves performance on tasks not yet trained.

Evaluation protocols are highly sensitive to data ordering, preprocessing, task construction, test-set design, task labels, and memory budgets. Continuum addresses this infrastructure by separating a raw **Dataset**, a **Scenario** describing temporal distribution changes, and a sequence of **TaskSets**. It provides loaders, transformations, task identifiers, logging, and metrics for class-incremental, domain-incremental, transformation-based, and online scenarios [2102.06253].

The scenario itself must be reported precisely. A result on Split MNIST may differ depending on class ordering, head structure, task identity, sample ordering, preprocessing, replay ratio, and whether evaluation is single-headed or multi-headed. Reproducible CL therefore requires more than publishing a model and optimizer; it requires specifying the complete stream and evaluation protocol.

The notion of performance should also be expanded beyond accuracy. Autonomous-agent research emphasizes retention of skills, adaptation speed, forward and backward transfer, sample efficiency, multisensory integration, uncertainty, autonomous task selection, behavioral competence, and performance over long horizons [1907.01929]. Resource-aware evaluation should include stored samples, model copies, generators, prompts, persistent state, floating-point operations, wall-clock time, energy, latency, and communication.

## 5. Applications and system-level continual learning

Continual learning is relevant wherever models encounter data after deployment, distributions change, and full retraining is impractical.

In autonomous agents and robots, the problem involves temporally correlated and uncertain sensory streams, embodied interaction, multimodality, unknown task boundaries, exploration, goal generation, and progressive skill acquisition. Developmental learning regulates how plasticity changes over time; curriculum learning organizes tasks from simpler to more complex; transfer reuses representations and skills; and intrinsic motivation guides exploration when external rewards are sparse [1907.01929].

In neural information retrieval, queries, documents, and relevance judgments arrive sequentially. Topic-controlled benchmarks such as Topic-MSMARCO show that topic shift and new-task volume affect embedding-based and pretrained rankers differently. Continual retrieval therefore requires architecture-aware choices among replay, regularization, and gradient constraints [2308.08378].

In supervised and unsupervised vision, continual learning includes classification, segmentation, detection, captioning, video understanding, image generation, VQA, person re-identification, and multimodal reasoning. The review literature identifies replay, regularization, distillation, generative replay, parameter isolation, and hybrid methods as the principal families, while emphasizing that conclusions from class-incremental classification do not automatically transfer to other vision problems [2109.11369].

A broader systems perspective treats continual learning as computationally constrained reinforcement learning. The objective is to maximize long-run average reward subject to limits on computation, memory, communication, energy, or processing rate. Under this view, forgetting is not intrinsically undesirable: retaining knowledge that will never be useful may reduce plasticity and future reward. The relevant question is what information should be retained, compressed, relearned, externalized, or discarded [2307.04345].

Recent system-level perspectives extend this view beyond parameter updates. Learning may occur during continual pre-training, post-training, or inference; it may be stored in parameters, memories, skills, tools, prompts, protocols, or external workflows; and it may be produced by off-policy gradients, on-policy reinforcement learning, model merging, zeroth-order optimization, evolutionary procedures, or direct memory writing [2608.06216]. This system-level formulation is particularly relevant to foundation models and agentic systems, where capability can migrate between internal parameters and external harness components.

## 6. Limitations, theoretical obstructions, and research directions

No continual-learning mechanism is universally effective. Replay requires storage, selection, processing, and often privacy management. Generative replay depends on generator quality and can accumulate errors. Regularization can preserve spurious features, become increasingly restrictive, or fail when tasks require substantial changes to shared parameters. Distillation depends on the overlap between current inputs and old-task variation. Parameter isolation increases capacity or routing requirements and may suppress transfer. Fixed representations provide stability but can lack plasticity for novel distributions.

Theoretical results make several limitations explicit. In nonlinear representation classes, there are task sequences for which no proper fixed-size continual learner can guarantee low error on all tasks, even with unlimited computation, because an early task may be compatible with multiple representations that become incompatible under later tasks [2203.14383]. In one-pass PAC continual learning, worst-case memory can grow as $\Omega(kdb/\varepsilon)$, including for improper learners. Multiple passes and improper aggregation can reduce this dependence, but proper learners retain stronger memory lower bounds [2204.10830].

The feature-selection perspective adds another obstruction: the learner cannot generally know whether a feature that is predictive on the current task will remain useful on future tasks. This makes continual learning partly a problem of delayed information and irreversible commitment, not merely parameter interference [2203.01012].

Important research directions include:

- **Compute-aware continual learning**: benchmarks should constrain and report memory, FLOPs, latency, energy, and time to convergence rather than treating sample storage as the only resource.
- **Large-model adaptation**: continual learning must address pretrained, modular, multimodal, and parameter-efficient systems whose update regime differs from small networks trained from scratch.
- **Open-world learning**: deployed systems need novelty detection, uncertainty estimation, active labeling, semi-supervised learning, self-supervision, and concept-drift detection.
- **Memory and capability management**: systems must decide what belongs in context, external memory, a skill library, a protocol, or internal parameters, and must support consolidation, compression, correction, and deletion.
- **Realistic autonomous evaluation**: benchmarks should include continuous streams, embodied interaction, unknown task boundaries, multimodal observations, sparse rewards, delayed consequences, long horizons, and changing environments.
- **Theory beyond fixed targets**: useful analyses should address tracking, recurrent tasks, relatedness, non-stationarity, computational constraints, and the conditions under which forgetting is beneficial rather than harmful.

Continual learning is therefore best understood not as a single algorithmic technique but as a family of strategies for managing knowledge under sequential information access and finite resources. Its central design problem is to determine what must remain stable, where plasticity should be allocated, how old evidence should be represented, and when previously acquired knowledge should be revised or discarded.

Source: https://www.emergentmind.com/topics/continual-learning